api.buttondown.email was inaccessible for 25 minutes between 20:09 and 20:34 UTC on July 23rd. api.buttondown.com — a separate hostname that serves the same API — and all other endpoints continued to work. The outage was caused by a broken DNS record after we migrated the API endpoints from Heroku to AWS: Pulumi deleted the api.buttondown.email record because of a bug that was fixed in the next version after the one we were running.
How did we detect the issue?
We set up alerting for both endpoints separately ahead of the migration, to make sure we caught a problem if it happened. Indeed, something happened. The alert went off at 20:18 UTC, but didn't page anybody, and was noticed by one of our engineers at 20:33 UTC.
How did we mitigate the issue?
At 20:34 UTC, we re-added the DNS record that was deleted, which restored the service immediately to users whose DNS providers doesn't cache failure responses.
How will we prevent this from happening again?
We're looking into our alerting setup to get notified more directly when something fatal happens. We're usually conservative because of alert fatigue, but this is a wake up call to make sure we better prioritize alerting.
We had hit the same Pulumi bug two weeks earlier and reported it; Pulumi shipped a fix quickly, but we hadn't upgraded yet. Going forward, when we know a dependency bug can affect us, we'll track the fix through to upgrade so we don't leave ourselves exposed.