Thursday, 2018-01-04

FoxyCart Service Unscheduled Downtime, AWS WAF Outage

2018-01-04 5:33pm EST: We just experienced 33 minutes of a service interruption. We are currently working to diagnose the cause, and will update here as soon as we have details.

UPDATE 5:58pm EST: We are still diagnosing, but at this point believe that the AWS ELB service had a major failure. AWS's status page hasn't been updated yet, but there are numerous reports of failures on Twitter. We use a few different ELBs and ALBs, and a failure with certain ELBs would be consistent with the behavior we saw (which was admin.foxycart.com and cdn.foxycart.com being impacted, but other domains remaining functional).

We will continue to monitor the situation, and are raising an emergency ticket with AWS, but everything is currently functioning properly.

UPDATE 6:19pm EST: We are still waiting for confirmation, but it does appear that this incident was isolated to a single ALB (Application Load Balancer) with the AWS WAF attached. Tweets suggest that the WAF + ALB may have had an issue on the AWS side of things. We also are curious if this may have been related to the recently released Intel memory security issue (and the corresponding mass reboot on AWS that happened earlier today).

FINAL UPDATE: AWS confirmed they had an issue:

2:45 PM PST Between 1:53 PM and 2:29 PM PST, we experienced connectivity issues for Web Application Firewall enabled Application Load Balancers in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.