On July 17th, from 13:52 to 15:19 UTC, some users were unable to log into Buttondown, and in some cases unable to subscribe/unsubscribe. This was caused by an upstream issue on Cloudflare.
How did we detect the issue?
Unfortunately, we learned about this issue from user reports quite late into the incident, which is unacceptable. We received user support tickets and social media posts of users unable to log in.
After testing various theories of why this issue might be happening (even within the team, not everyone was able to reproduce it, which made it even more complicated), we noticed the Cloudflare incident alert.
How did we mitigate the issue?
We had migrated a core component (our edge proxy) to Cloudflare a few days ago. Unluckily, if we hadn't done this, the incident wouldn't have happened. Luckily, it also meant that our old infrastructure in Heroku for doing the same thing was still active. We were able to forward traffic for buttondown.com back to Heroku at 15:25 UTC, which fixed the problem immediately (after clearing DNS cache.)
Here's a chart of unsuccessful logins during the incident:
How will we prevent this from happening again?
We'll start to automatically monitor % of successful logins as a metric, to make sure we can detect this earlier.
We'll continue to evaluate Cloudflare's stability in our infrastructure.