The dashboard and newsletter Archives for non-custom domain users (buttondown.com/[username]) were down for about 20min, from 16:46 to 17:06 UTC.
We introduced a code change with a new field to a database model which got deployed to our new infrastructure, while it was skipped from deploying to our old infrastructure, which is the one that applies the changes (migrations) to the database. As a result, the new infrastructure was left requesting a field that didn't exist in the database.
Notably, Archives hosted on custom domains were not affected as they are still served by our old infrastructure. Also, tracking links in emails were not affected as they don't interact with that part of the database.
How did we detect the issue?
We received automatic alerts from our monitoring that reported that most requests were failing with server errors, and we also received the specific errors regarding the missing field in Sentry.
How did we mitigate the issue?
We manually run the database migration so the field was created, immediately stopping the errors in the new infrastructure.
How will we prevent this from happening again?
We are fixing the bug in our deployment system that caused the database to not be migrated even when the code change was partially deployed.
We got paged way too late because while the alert went off, it was misconfigured and didn't immediately page a team member. Another alert paged them, but it took longer to trigger.