Saturday, 2020-05-16

Scheduled Maintenance: Database Upgrades, Downtime, scheduled 2 months ago

Date & Time: 2020-05-16, Saturday 11pm US Pacific Time (UTC -7)
2020-05-17, Sunday 2am US Eastern Time (UTC -4)
2020-05-17, Sunday 7am London (UTC +1)
2020-05-17, Sunday 4pm Sydney (UTC +10)
EveryTimeZone.com link

Duration: 30-60 minutes

Downtime: Yes. All requests to customer-facing cart, checkout, and receipt pages, as well as all admin access, will display a maintenance page.

Purpose: We are performing a major database upgrade, to allow us to continue to grow and scale into the future. We have performed this upgrade in our staging environment, and do not expect any issues. We will, however, have all systems ready to revert to the previous database infrastructure if things go sideways.

Resolution:

UPDATE 11:25pm Pacific Time: We resumed service a few minutes ago, and all systems look perfect. We will continue to monitor to ensure there aren't any unpleasant surprises.

UPDATE 2:57am PDT: We have identified an issue with historical dates being displayed 7 or 8 hours ahead of when they should display, in the admin and API. We are working to resolve this.

UPDATE 4:15am PDT: Transactions between May 7th and the update window tonight are impacted. We are working to correct the data.

UPDATE 10:40am PDT: We have corrected some of the data, and are just working to get the rest updated. We'll update again here (and by email to impacted users) when all data is corrected.

UPDATE 1:00pm PDT: We have updated all transaction dates and transaction log dates. We are continuing to work on some additional tables.

UPDATE Thursday May 21, 2020, 12:25am PDT: All data has been corrected.