UPDATE: New time.
Date & Time: Wednesday June 23, 3am CST (everytimezone.com)
Duration: 1 hour.
Purpose: We'll be upgrading our production environment with the latest security patches, as well as updating a webserver configuration to prevent occasional errors during code deploys.
Additional Information: As part of this upgrade, we will be upgrading OpenSSL to the latest version. Though we consider it extremely unlikely, if you're using a very old SSL client on your server, you may have difficult connecting to our systems via the API. (There should be no impact at all to browsers.)
Resolution: We will add a note here when the maintenance window has closed.
Root Cause Analysis: As part of the scheduled security upgrades, a patch to part of the Linux kernel was included. Unbeknownst to us, this specific system, when updated, requires locales to be reinstalled.
Though our deploy automation does indeed automate locale installation, the timing of the automation and the upgrade was such that the following happened.
Unfortunately, PHP wasn't restarted after the locales were reinstalled, which resulted in the 50 minutes of locale issues.
We are updating our automation to ensure PHP is not restarted if the required locales aren't present (step #2 above), and to ensure that PHP is restarted in the event additional locales are installed during the automation.