| The RatioA weekly newsletter on reliability economics | |
|
The Number
19% and 17% (AI for investigating incidents and summarizing logs)
Companies are buying AI to clean up failures faster, not to stop them
For every dollar preventing failures, how much goes to cleanup? That tilt is the ratio. Teams using AI are tilted twice as far toward cleanup -- because they bought it for post-incident investigation (19%) and log summarization (17%). Just 13% use it to catch problems early.
The Ratio Take
Teams reaching for AI were already drowning, they bought it because they were struggling. But they're spending on the wrong side: faster cleanup, not fewer failures. Summarizing outages faster isn't a reliability investment. It's more ambulances, not better roads. The organizations spending most on reliability AI are already losing most to failures, and spending it on faster cleanup, not fewer fires.
|
|
This Week in Reliability
AI Agents Need Guardrails
Autonomous AI agents are shipping to production with code-execution privileges but without the observability, policy enforcement, and audit infrastructure that took traditional software decades to build. Teams are now scrambling to retrofit control planes around agents deployed for speed.
|
| 1 | Cognitive Today · Primary evidence Article argues for integrating statistical guardrails into MLOps pipelines to prevent catastrophic failures in LLMs and reinforcement learning systems, treating guardrails as part of continuous AI reliability engineering rather than afterthoughts. The Ratio Take The MLOps community finally realized what SRE learned in 2010: you can't bolt safety onto a system after it's in production and expect it to work. Guardrails belong in the pipeline, not in the incident response playbook. The real cost is the production fires you fight while waiting for the culture to catch up to the tooling. Guardrails in CI/CD, not in the war room. |
| 2 | GitHub (ai-boost) · Ecosystem response Community-curated repository collecting tools and patterns for AI agent harness engineering including authentication (Nango), diagnostic guardrails (AgentDoG achieving 91.8% accuracy), observability, permissions, and orchestration across 700+ APIs. Frames harness engineering as system-level design for agent execution environments. The Ratio Take The gap between 'agent that demos well' and 'agent you can run in production' is now large enough to need its own GitHub awesome-list. This is the missing control plane—authentication, observability, guardrails—that every production system requires but AI agents shipped without. If you're budgeting for agent deployment, budget for the harness too. The agent awesome-list is a reliability gap made visible. |
| 3 | Codiste · Taxonomy/framework Defines harness engineering as system-level design of execution environments, control loops, state persistence, and guardrail infrastructure enabling agents to work reliably across sessions. The Ratio TakeHarness engineering is the tax you pay for shipping agents without a runtime—the infrastructure layer between 'it works on my laptop' and 'it runs in production without setting the building on fire.' Agents need runtimes, not just prompts. |
| 4 | The New Stack · Unintended consequence Developer discovered his own code integrated into OpenClaw agent framework without attribution, raising questions about agent security, code provenance, and the supply chain risks of autonomous systems that pull and execute code. The Ratio TakeWhen your agent can autonomously pull dependencies, you've outsourced supply-chain security to an LLM with no audit log. Agent supply chains are unaudited by design. |
| 5 | The New Stack · Adjacent—AI infrastructure Argues that AI retrieval at production scale requires systems-level thinking—connection pooling, caching strategies, failure modes—rather than just selecting the right vector database or embedding model. The Ratio TakeRAG hit the same wall every other technology hits at scale: the tooling is easy, the distributed systems part is hard, and nobody budgeted for the second part. Retrieval at scale needs SRE, not better embeddings. |
| 6 | Lorin Hochstein · Organizational failure mode Critiques the trend of cutting middle managers and increasing direct reports per manager under the assumption that AI can replace coordination work, warning that coordination is complex sociotechnical labor that AI tools cannot yet replicate effectively. The Ratio TakeIf you think AI agents can replace the humans who coordinate incident response, policy reviews, and cross-team architecture decisions, you're about to learn an expensive lesson about what 'coordination' actually means. AI can't coordinate; it can only automate scripts. |
| 7 | StackGen · Measurement innovation Proposes 'Negative MTTD' (mean time to detect before the incident happens) as a key SRE metric for the agent era, measuring how far in advance systems can predict and prevent failures rather than just detecting them faster. The Ratio TakeThe agent reliability gap creates demand for a new metric class: how long before the agent breaks something do you know it's going to break? Agents force us to measure prediction, not detection. |
|
The Challenger — Comment of the Week
"Our AI assistant now summarizes every postmortem. Engineering loves it. I realized last week that nobody has actually verified whether the summaries match the real timeline. They sound authoritative, so we assumed they were right."
— Comment from an SRE Lead, Tech Finance Company
1 in 5 teams now uses AI to investigate incidents. Polished output starts to feel like correct output. Aviation researchers have a name for this: automation bias. Confident-sounding results suppress the instinct to verify.
A hallucinated root cause in a clean summary can misdirect the entire postmortem. The real failure repeats quietly.
Treat your AI summarizer like any other dependency. Give it an error-rate SLO. Spot-check 10% of outputs per sprint against the raw timeline.
Unvalidated AI in the postmortem loop is a monitor with no monitor. Closes a blind spot
|
|
The Ratio is a weekly newsletter by Florian Hoeppner. Early edition, you are one of the first.
Forwarded this? Get The Ratio in your inbox every week.
|
Take the assessment → reliabilityeconomics.com/benchmark Reply to this email with your take.
How this newsletter gets better: Every link is tracked and every vote is counted. Sections that get 👎 get replaced. Topics that get 👍 stay. No personal data is stored — just click and vote counts per section. Your feedback shapes the next edition.
|
|
|