On May 8th, our tracking domains (buttondown-0000.com through buttondown-0009.com) experienced downtime from around 20:30 to 21:42 UTC, which caused the links in emails from newsletters with tracking enabled to be unavailable.
This was caused by Heroku resetting the DNS targets for these domains, making the DNS targets previously set invalid. This made the TLS certificates for these domains to be invalid, which meant that Heroku couldn't serve HTTPS traffic to users.
How we detect the issue?
Unfortunately, our automated systems weren't monitoring tracking domains specifically, so we were alerted by user reports instead of automated systems. This is not acceptable, and we have taken steps to actively monitor all endpoints, not just our main ones.
How did we mitigate the issue?
At 21:07 UTC we set the new DNS targets for the domains and requested Heroku to refresh the ACM certificates intending that it would emit new, valid certificates. While this worked for buttondown-0009, -0006 and -0003, it didn't seem to be working for the rest of the tracking domains. This might've been related to the Let's Encrypt downtime that was happening around the same time.
At 21:39 UTC we deployed a mitigation for the rest of the domains by leveraging the Cloudflare proxy to handle HTTPS instead of Heroku. This worked immediately and restored access to all domains.
How will we prevent this from happening again?
- We're improving our monitoring to notice these issues sooner, including traditional uptime monitors but also TLS certificate monitoring and metric analysis to detect unexpectedly low traffic.
- We're working on better internal documentation to act faster if/when these type of issues happen again.
- We haven't been able to identify why Heroku reset the DNS targets for the domains, but we've contacted them for more information. Our best guess is a Heroku issue, potentially related to our use of CNAME flattening in those domains.
- We were already looking into migrating our domain ingress to another platform since before this incident, and we will be prioritizing these efforts to replace Heroku as our main hosting provider.