Tuesday, 2020-04-21

FoxyCart Service Unexpected Service Interruption: Hardware Failure in Cache Cluster

At about 6:45am Pacific Time, our customer-facing functionality (cart, checkout) experienced an issue that resulted in about 9 minutes of downtime. We are still working to determine the cause. We will update here as we have additional details.

UPDATE 7:13am: We have revised the impact window, as it looks like it started at about 6:45am Pacific Time. We have narrowed down the cause to in issue connecting to our Redis cluster. We're still working to determine the cause.

DIAGNOSIS, UPDATE 7:57am Pacific Time: We have confirmed that our AWS Elasticache cluster had a hardware failure on our primary writer node. This was handled automatically, but did take about 15 minutes. We had already been exploring possible alternative approaches to handle our cache cluster, and we will continue to explore those options.