Wednesday, 2019-12-18

FoxyCart Service Unplanned Service Interruption

We are exploring the cause of the downtime. We'll update as soon as we have more details.

7:22am PST: We're back. It appears we had a misconfigured load balancer that didn't properly respond to a traffic spike, but we're working to confirm.

7:28am PST: We are continuing to experience service issues. We're still working on restoring service.

7:54am PST: We restored service at 7:32am PST and have been monitoring. We are continuing to investigate the root cause of this issue.

3:25pm PST: Though we are still working to discover the root cause, we have identified an issue that prevented our systems from scaling as intended. (Or rather, scaling occurred, but a bottleneck prevented the new resources from being helpful.) We have system improvements planned and currently being tested, but in the meantime we did identify an improvement to ameliorate the issue we had with scaling today.