Public postmortem: duplicate sends

(We've historically published postmortems on our status page, but this is annoying for two reasons: discovery and accessibility. Discovery because it's not easy to find, and accessibility because it's not easy to read. We've ported those over here, and all new postmortems will be published here from now on.)

On Thursday, October 16th, we incorrectly sent an extremely high number of duplicate emails for a total of five newsletters from 10:39am to 10:47am. This was caused by a bug in (ironically) our rate limiting logic, which was meant to rate limit at a domain level. (All five users have been notified.)

How did we detect the issue?

Two specific alerts triggered: one alerting us that an email had recorded multiple duplicate deliveries (i.e. "justin@gmail.com received this email more than once!"), and another alerting us that the total number of events for an email was significantly higher than the number of subscribers ("The estimated subscriber count for 'Hello, world!' was 100, but we have 1000 events!").

How did we mitigate the issue?

We immediately turned off rate limiting for all newsletters via a flag, followed by a roll-back of the problematic code. Out of an abundance of caution, we paused sending altogether (from 10:47am to 11:00am) before resuming sending at 11:00am. A fix was quickly identified and deployed at 11:02am, at which point we deemed the incident resolved.

How will we prevent this from happening again?

Two ways: one object-level and one system-level.

1. Our testing for this part of the sending logic was missing some critical bits of logic; we've added those, and going forward will be extremely sensitive to edge cases of this nature.
2. At a high level, Buttondown sends emails by calculating all subscribers that should receive it, batching them into groups of N, and then filtering them down to M subscribers for various reasons (deliverability; rate limiting; etc.). It seems silly to say, but the way this bug manifested was that M was larger than N, an obvious invariant violation. We've added a check for this, as well as a host of other sanity checks along the way in this part of the SMTP pipeline.