From the evening (US time) of Tuesday June 18 through mid-day Wednesday June 19, 2019, our email service provider SendGrid.com had an issue that resulted in delays and dropped emails. This resulted in some emails not being delivered to Foxy admins and their customers. The following text (slightly modified) was emailed to store email addresses that had transactions during the impacted window.
What happened?SendGrid's status page is light on details, but some emails were delayed, and some of the delayed emails were effectively lost. Though we do have logs, due to the nature of the failure, getting individual emails that weren't delivered from the logs would prevent us from getting you a quick notification of the problem.
What do you need to do?
If you rely on emails alone to trigger shipping/fulfilling products, please login to your Foxy account and check your orders to make sure you didn't miss any.
You can also resend email receipts from the Foxy admin. Find the transaction you'd like to resend, and click the "email" link below the customer's name.
Further, if you have your own SMTP servers you'd like Foxy to use to send mail, you can configure that in the Foxy admin. Documentation is here.
How will this be prevented in the future?
We are exploring moving from SendGrid to AWS SES or other providers to handle our emails. We also have been exploring services to better monitor our email deliverability, though in a situation like this (where only a small fraction of our emails are lost), monitoring can be tricky.
Though we've been with SendGrid for many, many years, this isn't the first time they've "lost" emails. Because our email deliverability is so high, we're approaching this slowly, but we do expect to make some changes in the coming months. No changes will be required from our users when this happens.
FoxyCart ServiceSubscription Past Due Data Error
While doing database maintenance, a database query was performed incorrectly, resulting in bad data being written / overwritten to subscriptions' past due amounts, and in some cases the first_failed_transaction_date value.
We are currently working to restore the data. We will post updates here as they are available.
Update 3:20pm PST: We are still working on restoring and correcting data, but all subscriptions without a first_failed_transaction_date have had their past_due_amount values reset to 0 for now.
Update 5:30pm PST: We're prepping to restore the data.