The RatioA weekly newsletter on reliability economics | | |
|
The Number
1 of 57
Only 1 of 57 Technology companies in the benchmark has its reliability investment calibrated near-optimally — 29 under-invest and 27 over-invest.
Technology is the largest sector in the benchmark at 57 responses. 29 are classified as under-investing in reliability. 27 as over-investing. Only 1 is near-optimal.
Neutral
Tech companies aren't failing to invest. They're failing to aim. A near-even split between under and over-investment, with one company in the middle, tells you the sector has a calibration problem, not a spending problem. Think of it like a factory floor where half the machines run too hot and the other half too cold, and exactly one thermostat works. The money is there. The dosage is wrong in both directions.
| 56 of 57 technology companies are either starving their reliability budget or drowning in it. One got it right. |
|
|
This Week in Reliability
AI Agents Hit Production Reality
The first wave of autonomous AI agents is moving from demos to production infrastructure, forcing teams to confront new operational risks—agent observability, failure modes, and the cost of debugging multi-stage workflows—that existing SRE tooling wasn't built to handle.
|
Surfing Complexity · Primary evidence Lorin Hochstein argues that AI agents are now capable of handling significant on-call work, citing Boris Tane's position that on-call is becoming theatre as agents take first-responder duties. The post explores how AI will change incident response workflows and the reliability challenges this introduces. Neutral This is the reliability economics pivot point: when agents start changing production state autonomously, your incident taxonomy breaks. You're not budgeting for 'agent decided wrong' as a root cause category, but you will be. | Agent autonomy creates failure modes your runbooks don't cover. |
|
Grafana Labs · Vendor response—agent context problem Grafana tested how well AI agents can debug full-stack incidents and measured performance improvements when agents have access to Grafana Cloud's Knowledge Graph versus operating without that context. The post demonstrates that agent effectiveness in RCA depends heavily on contextual infrastructure data. Prevention Here's the hidden agent tax: graph infrastructure. Agents need semantic context about your stack to avoid expensive hallucinations during incidents, which means new platform investment before agents deliver ROI. This shifts preventive spend forward in ways most roadmaps haven't accounted for. | Agent RCA quality depends on context infrastructure you don't have. |
|
Grafana Labs · Infrastructure scaling for agent observability Grafana shares capacity planning and load testing lessons for running Alloy as a centralized telemetry gateway handling enterprise-scale volumes—tens of millions of active series, terabytes of logs daily, and tens of thousands of trace spans per second. Firefighting Agents need telemetry pipelines that can handle both human-generated and agent-generated observability data at scale—double the cardinality problem. | Agent telemetry doubles your pipeline capacity problem overnight. |
|
CNCF Blog · Prerequisite for agent RCA Atlassian describes the challenge of correlating hundreds of interconnected microservices across multiple regions during incidents, where telemetry volume overwhelms human operators and finding causal factors requires automated multi-signal correlation. Prevention Multi-signal correlation is table stakes for agent-assisted RCA—this is the detection gap agents are supposed to close, but it requires investment in correlation infrastructure first. | Agents inherit your correlation debt before fixing it. |
|
Gremlin · Agent safety prerequisite The post explains how Kubernetes' minReadySeconds parameter makes new pods prove they can stay ready before a rollout proceeds, and provides guidance on setting it and identifying workloads that lack this safety control. Prevention When agents start triggering rollouts, this config line becomes a reliability guardrail—the difference between an agent-driven deployment that self-heals and one that cascades. | Agent-driven rollouts expose every missing config guardrail. |
|
Surfing Complexity · Real-world agent operational risk signal GitHub experienced another major incident lasting nearly eight hours, with a public writeup posted unusually quickly. The post is the latest in Hochstein's ongoing series analyzing GitHub's operational challenges. Firefighting GitHub's incident cadence is a preview: as more companies deploy agents that touch production, expect your incident frequency to spike before it drops—agent learning curves cost MTTR budget. | Agent adoption raises incident frequency before it lowers it. |
|
Nobl9 · Vendor product-ization of agent workflows Nobl9 released Nobl9 Assist, an in-context AI assistant that answers plain-language questions about SLO burn, budget drain, and feature configuration using live organizational data. It can draft and apply configuration changes with explicit user approval. Neutral In-product AI assistants mark the shift from 'agents as tools' to 'agents as coworkers'—you're now budgeting for agent UX, agent training data pipelines, and agent approval workflows inside every platform purchase. | Agent interfaces become a new platform buying criterion. |
|
|
The Challenger — Career Spotlight
Mid-size payments platform. 40-person platform org. Starting prevention-to-firefighting ratio: 0.71. Eleven agents running in production, and nobody could say what any of them had done.
Twelve weeks. Tooling came second. First they made every agent action emit a decision record: input, tool call, permission used, outcome. Then they cut agent scope back to what those records justified.
Ending ratio: 1.34. Agent-caused incidents had been roughly a third of the total. Nobody had attributed them there. Nothing was added to the budget.
Prevention
| You cannot rebalance what you cannot attribute |
| Should this replace the favorite? | |
|
|
The Ratio is a weekly newsletter by Florian Hoeppner. Early edition, you are one of the first.
Forwarded this? Get The Ratio in your inbox every week.
|
Take the assessment → reliabilityeconomics.com/benchmark Reply to this email with your take.
How this newsletter gets better: Every link is tracked and every vote is counted. Sections that get a down vote get replaced. Topics that get an up vote stay. No personal data is stored — just click and vote counts per section. Your feedback shapes the next edition.
|
|
|