Public postmortem: manually adding subscribers failed

What happened, and why

Buttondown's author-facing dashboard — the app you log into to send emails and manage subscribers — uses Buttondown's external-facing API for the vast majority of its work. As part of a broad initiative to tighten up our API schema — to improve safety for, ironically, exactly things like this, and to make it easier for new users to start working with our API — we've shipped a number of changes you can think of as tightening the language of the API: the functionality doesn't change, but the contract gets stricter about what you give us and what we hand back. One such change, which we rolled out on June 13th (UTC), was to reject as invalid any API request that sends us extraneous data. This is generally considered a good idea, because you don't want to give callers the false impression that anything they pass in is meaningful. Unfortunately, we turned out to be one such caller: our manual subscriber-creation flow passed in an extraneous field — a client-only active toggle — which, whenever it was present, caused the request to fail.

How we discovered it

The tricky thing about this genre of issue is that it also has a common and completely reasonable explanation: someone running an older version of the front-end bundle that talks to our app. If we remove a field from the API contract and deploy that change, a user who hasn't yet refreshed their browser to pick it up can send an API request containing the old field, which then comes back as a 422. We have high-level logging for this specific status code, but it was deliberately insensitive and didn't trip for something like this — a comparatively low-volume, manual path that was still succeeding some of the time.

Sadly, that meant we discovered it through customer support tickets: the worst possible way to find out.

The change was also far too broad to catch by hand. "Reject all undeclared fields, everywhere" can only be fully verified by exercising every workflow that writes to the API — not something you can spot-check, which is how a broken flow reached production unnoticed.

How we mitigated it

On June 15th at 14:04 UTC we merged and deployed a fix that builds the subscriber-create request body from an explicit list of API fields instead of spreading the whole form object, so client-only fields can no longer leak into the payload, along with a regression test that asserts the request body never contains active. App-sourced 422s on the endpoint dropped from roughly 20/hour to near-zero within the half hour, by ~14:30 UTC. In total, 454 author attempts to add a subscriber failed over the window.

A small tail of failures continued through the evening: authors who still had the old, broken bundle loaded in their browser kept hitting the error until their page next reloaded and picked up the fix — the same version-skew effect described above, now working in our favor as the stale bundles aged out.

What we're doing

1. We've added alerting for app-sourced 422s. A new checker watches first-party (source=app) requests over a settled window and alerts when an endpoint crosses a small threshold of validation failures, grouped by route and annotated with the exact rejected fields (e.g. body.payload.active). A 422 on a first-party request almost always means our own frontend sent a body our own backend rejected — that's now something we catch in hours, not days. Third-party API 422s are intentionally excluded, since those are the caller's responsibility.
2. Our frontend tests now run against a strict, stateful mock of the API instead of a permissive one that accepted any payload. The whole reason this slipped past code review and CI is that our test doubles were more forgiving than the real backend — the mock happily accepted the active field the real schema rejects. A strict mock closes the gap the type system structurally can't: a test that exercises a form now fails if the form sends a field the API doesn't declare.
3. We're making the frontend send explicit, whitelisted request bodies rather than spreading form state into payloads, so a client-only field can't silently become an API field. The subscriber form is fixed; we're auditing the remaining write paths for the same pattern.
4. We're ramping broad, behavior-changing rollouts instead of flipping them globally. Changes like "reject all undeclared fields" alter behavior the OpenAPI contract can't express, so type-safety gives false confidence. Those go out gradually, one endpoint at a time, with 422 rates watched as a guardrail before they reach everyone.

Lastly, we're elevating manual subscriber creation to a core flow, so we invest in testing and rollout work around it the way we already do for things like sending an email or logging in.