From 22:41 UTC on October 20th to 00:53 UTC on October 22nd, Archives on about 140 custom domains failed to load with a TLS error (ERR_SSL_VERSION_OR_CIPHER_MISMATCH). A few Archives on root domains (like example.com) and on domains proxied through Cloudflare were down for longer. Email sending, the dashboard and Archives on buttondown.com were not affected.
As part of our migration off of Heroku, we moved custom Archive domains from Heroku to a CloudFront distribution. This works by giving each custom domain its own CloudFront "tenant" with its own TLS certificate, and then pointing custom-domains.buttondown.com (the CNAME target authors set up) at CloudFront. Before switching DNS, we created tenants for every domain we had marked as valid and checked that all 212 of them worked through CloudFront. They did, so we switched DNS at 22:41 UTC.
The problem was the domains that weren't on that list:
- Domains we had never verified. We only check a custom domain's DNS when its settings page is loaded, so domains whose settings page hadn't been loaded since they were added were left in an unverified state. Heroku served them anyway, so this never mattered before. The migration only created tenants for verified domains, so 146 domains with perfectly fine DNS ended up pointing to CloudFront without a certificate.
- Root domains using CNAME flattening. DNS doesn't allow CNAME records on root domains, so some providers (like Porkbun) "flatten" them by publishing our IP addresses instead. CloudFront can't verify ownership of a domain that way, so it never issued their certificates.
- Domains proxied through Cloudflare. Same as flattening, Cloudflare's proxy hides the CNAME record from CloudFront, so it couldn't issue their certificates either.
How did we detect the issue?
Unfortunately, we found out about this issue from user reports, which is unacceptable. The first report came in at 02:14 UTC on October 21st, outside of support hours, and it reached an engineer at 14:28 UTC. The first report about the unverified domains came in at 19:32 UTC, and when we tried it the domain loaded fine for us, which delayed things further.
The checks we did during the migration only covered the domains we had migrated, so they all passed.
How did we mitigate the issue?
For the never-verified domains, we verified them and created their CloudFront tenants manually, which restored them as soon as their certificates were issued.
For root domains using CNAME flattening, we asked authors to add a _cf-challenge DNS record, which lets CloudFront verify the domain, and started showing it in the custom domain settings. The permanent fix was attaching static IPs to our CloudFront distribution, which lets CloudFront verify these domains without the extra record. This took until October 27th because we hit our AWS quota for static IPs and had to wait for AWS to approve an increase.
For domains proxied through Cloudflare, we reached out to the authors with the DNS changes they needed to make.
Timeline
| Time (UTC) | Event |
|---|---|
| Oct 20, 22:41 | custom-domains.buttondown.com is switched to CloudFront. |
| Oct 21, 02:14 | First user report, about a root domain using CNAME flattening. |
| Oct 21, 14:28 | An engineer starts investigating. |
| Oct 21, 15:00 | We try to set up static IPs for CloudFront and hit our AWS quota. |
| Oct 21, 15:40 | The first affected root domain gets its certificate after the author adds a _cf-challenge record. |
| Oct 21, 16:24 | _cf-challenge.custom-domains.buttondown.com is published. |
| Oct 21, 19:32 | First user report about a never-verified domain. |
| Oct 21, 23:53 | Reports about never-verified domains are escalated to engineering. |
| Oct 22, 00:29 | We start creating CloudFront tenants for never-verified domains manually. |
| Oct 22, 00:53 | 140 of those domains have their certificates and work again. |
| Oct 22, 13:54 | Custom domain settings show the _cf-challenge record when a root domain needs it. |
| Oct 22, 16:20 | Hourly verification of never-checked domains is deployed. |
| Oct 22, 19:02–19:30 | We reach out to authors whose DNS needs changes on their end. |
| Oct 22, 21:45 | Another root domain gets its certificate. |
| Oct 27, 20:53 | Static IPs are attached to the CloudFront distribution. |
| Oct 27, 21:26 | The last two root domains get their certificates. |
How will we prevent this from happening again?
We now verify never-checked domains every hour (deployed at 16:20 UTC on October 22nd), instead of waiting for someone to load the settings page.
Root domains using CNAME flattening now work out of the box thanks to the static IPs.
When planning future migration work, we'll build the list of affected domains from DNS instead of from our database, so we don't miss domains that our database doesn't know about.
We'll add monitoring for TLS errors on custom domains, so we find out about these issues before our users do.