The deploy that took down the checkout flow for 45 minutes, the postmortem that followed, and the lesson about feature flags that stuck.

#Production #Incident #Postmortem

The Incident

My first production incident was a checkout flow that broke for 45 minutes on a Tuesday afternoon. The cause was a change I'd deployed that morning — a "small" refactoring of the payment processing logic that I'd tested locally and in staging without issue. In production, however, the change interacted with a specific payment provider's API in a way that my tests hadn't covered, and every checkout attempt that used that provider failed silently — the customer saw a generic error message, but the payment was never submitted. We discovered the issue not from monitoring but from a customer support ticket, which is the worst way to discover a production issue because it means real customers have already been affected.

The rollback took 10 minutes once we identified the deploy as the cause, but identifying the deploy as the cause took 30 minutes because the error wasn't obvious — the checkout appeared to work from the user's perspective, and the error only manifested on the backend. The 45-minute total downtime wasn't catastrophic, but it felt catastrophic to me, because I had broken something that real customers were trying to use, and the company was losing revenue for every minute the checkout was down. The senior engineer who led the response was calm and methodical — "we'll fix this, then we'll understand why it happened, then we'll prevent it from happening again" — and that calmness was the most valuable thing I took from the experience, because panic makes everything take longer.

The Postmortem

The postmortem was blameless, which is the only kind of postmortem that actually improves the system. The question wasn't "whose fault was this?" but "what about our process allowed this to happen, and how do we change the process so it can't happen again?" That framing matters because blame makes people defensive, and defensive people don't contribute honestly to the analysis. The postmortem identified three process failures: the test suite didn't cover the specific payment provider's API behavior, the monitoring didn't alert on the silent failure, and the deploy wasn't staged behind a feature flag that could have been turned off without a rollback.

The fix that had the biggest long-term impact was feature flags. After this incident, every significant change was deployed behind a feature flag — the code was deployed to production but not active, and a flag controlled whether it was used. That meant a problem could be fixed by turning off the flag, which took seconds, rather than rolling back the deploy, which took minutes and had its own risks. Feature flags transformed our deploy safety: instead of "deploy and hope," we had "deploy, verify, activate, and if something goes wrong, deactivate instantly." That pattern has become the default for every team I've worked on since, and it all started from this incident.