The RatioA weekly newsletter on reliability economics | | |
|
The Number
2 of 101
Only 2 of 101 organizations in the benchmark spend more than half their reliability budget on prevention, the other 99 spend the majority reacting to failures after they start.
The benchmark asked organizations how much of their reliability spend goes toward prevention. 23 spend under 10%. 26 spend 10–24%. 50 spend 25–49%. Just 2 spend 50% or more.
That's 99 of 101 organizations spending most of their reliability budget reacting to failures, not stopping them.
Firefighting
Every SRE framework says shift left, invest in prevention. Almost nobody does. 2 out of 101 is not a rounding error. It's a structural reality.
This is the fire department problem in building codes. Everyone agrees prevention is cheaper than response. But fire departments get funded after the fire, not before. Prevention budgets compete with visible, urgent work, and they lose. Every sprint, every quarter.
Any team that crosses the 50% prevention mark has essentially no competition at that level of practice. Not because others can't get there. Because nobody is trying.
| 99 of 101 organizations spend more on cleaning up failures than preventing them. |
|
|
This Week in Reliability
AI Agents Need Harness
From Google SRE to startup launches, the industry is converging on a single problem: AI agents can code and deploy, but without runtime harness—guardrails, observation, rollback—they shift cost from development into operational chaos. The reliability economics question is no longer whether to adopt agents, but how much you'll spend cleaning up after unsupervised ones.
|
sre.google · Primary evidence Google SRE published a new resource on reliable operations for AI engineering workloads, detailing practices for managing AI model deployment, inference reliability, and operational patterns specific to AI systems at scale. Prevention When Google SRE writes a named practices document, it's because the pattern has matured past experimentation. This isn't about training infrastructure—it's about productionizing agent workflows with the same rigor Google applies to Search. The implication: AI operations are now a distinct reliability domain with predictable failure modes and measurable prevention ROI. | Google SRE formalizes AI ops as a distinct reliability domain. |
|
Not A Factory Anymore · Historical precedent The article argues we're at an inflection point for AI tooling similar to when Jenkins emerged—lots of custom scripts, no standard deployment pattern, teams reinventing CI/CD independently. AI agents today lack shared runtime infrastructure, observability hooks, or rollback conventions. Neutral This is the reliability economics framing the industry needs: every team building custom agent harness is duplicating prevention spend. The Jenkins parallel matters because it predicts consolidation—someone will ship the standard runtime, and laggards will carry double the operational cost. The question isn't if you'll adopt harness patterns, but whether you'll pay to invent them or buy them off the shelf. | Custom agent harness duplicates prevention spend across every team. |
|
The New Stack · Vendor response Dynatrace announced autonomous SRE agents that integrate with its observability platform to identify and remediate issues in AI operations workflows, focusing on the complexity of distributed inference and model serving. Firefighting Vendor response to the harness gap—if you can't see what agents are doing in production, you can't budget for cleanup. |
InfoWorld · Adjacent signal Article covers how AI tools (copilots, automation) are reducing toil and accelerating incident resolution for SRE teams by correlating signals across distributed systems and automating repetitive operational tasks. Firefighting AI for SREs vs. SRE for AI—this is the reactive half of the story, speeding up war rooms instead of preventing fires. |
Not A Factory Anymore · Adjacent pattern Article argues that complex engineering programs fail when teams defer hard decisions, unresolved dependencies, and untested assumptions to later phases, resulting in perpetual '80% complete' status and compounding rework cost. Prevention Agent deployments without upfront SLO/rollback/observability alignment will hit this wall—you can't retrofit harness into a live agent swarm. |
The New Stack · Vendor response Pilot Protocol is a new open framework designed to standardize how AI agents interact with infrastructure, coordinate across services, and integrate with existing DevOps toolchains, positioning itself as the 'HTTP for agents.' Prevention This is the Jenkins contender—if Pilot Protocol gains traction, it becomes the standard harness layer and everyone else pays integration tax. |
Cloudflare · Adjacent infrastructure gap Cloudflare released pvcli, an open-source debugger for privacy-preserving protocols (Oblivious HTTP, Private Access Tokens) that Apple and Microsoft use, explicitly designed to help developers troubleshoot agent-to-service interactions that go through privacy layers. Neutral Agent harness isn't just observability—it's debuggability through privacy boundaries, which nobody budgeted for in 2025. |
|
The Challenger — Career Spotlight
A fintech SRE team, 8 engineers in a roughly 600-person org, opened 2023 at a 0.6 prevention-to-firefighting ratio. Eighteen months later: 2.1.
The lever wasn't headcount. It was schedule. On-call rotation now owns the SLO dashboard review in weekly planning. Prevention work gets a standing slot instead of losing to features every sprint. That alone changed backlog priorities.
Toil tracking surfaced three recurring alert categories eating 40% of response hours. They automated two. Moved the ratio more than any hiring round had.
Prevention
| Schedule changed the ratio. Headcount didn't. |
|
|
The Ratio is a weekly newsletter by Florian Hoeppner. Early edition, you are one of the first.
Forwarded this? Get The Ratio in your inbox every week.
|
Take the assessment → reliabilityeconomics.com/benchmark Reply to this email with your take.
How this newsletter gets better: Every link is tracked and every vote is counted. Sections that get a down vote get replaced. Topics that get an up vote stay. No personal data is stored — just click and vote counts per section. Your feedback shapes the next edition.
|
|
|