There Is No Root Cause

Working on AI products, I keep running into the same interesting realisation: the tried and true methods of root cause analysis that I've relied on forever don't work anymore. The trail that used to lead back to a line of code, a config change or a bad deploy now tends to run out somewhere much less simple, usually at the words "the model did it". Thankfully, there are many people much smarter than me already discussing this problem, and over the last couple of months I've been reading as much of it as I could find:

• AI Incident Retrospectives: When 'The Model Did It' Is the Root Cause - Tian Pan, and the piece that reframed the question for me
• Observability for AI Agents: Logging, Tracing, and Debugging Autonomous Workflows - from MightyBot, a company that builds AI agents for regulated industries such as lending and insurance, where every automated decision needs an audit trail. It's a vendor blog, but it has the clearest practical description I found of debugging something that doesn't behave the same way twice
• Causal Software Engineering: A Vision and Roadmap - Pietrantuono, Giamattei, Russo, Siebert and Walkinshaw, an academic paper that turned out to be more useful to a product manager than I expected
• How Anthropic's product team moves faster than anyone else - Lenny Rachitsky with Cat Wu, Head of Product for Claude Code
• the history of two Toyota-era quality tools, the Five Whys and the Ishikawa diagram, which I went back to once I realised the whole conversation was borrowing from them
• and a pile of AI observability tool round-ups, which were thorough about what each platform can trace and almost silent on what you should do with the trace once you have it

I was obviously hoping to find a replacement for the same kind of methodology: an incident method adapted for AI, ready to pick up and use. Instead, there was a much clearer explanation of why the old methods fail. Part of the problem is the phrase "root cause" itself, which assumes a failure has one cause and will happen again if you repeat what triggered it. And one of the diagnostic tests I recommended earlier this year leans on exactly that assumption.

The test I recommended

In Triage, the Layer 3 article in my AI Fluency series, I called the reproduction test the single most useful diagnostic question in AI product work. Take the exact query that failed, send it straight to the raw model with the right context pasted in by hand, and see what happens. If the failure reproduces, it's a model problem. If it disappears, your pipeline is at fault.

I still think that test is useful. But it asks a yes-or-no question: does the failure happen again? My further research kept pointing out that for AI systems, the more useful question is how often it happens.

Tian Pan puts the problem as:

The 'bug' doesn't reproduce. You can't point to a line of code. The failure occurred probabilistically, somewhere inside billions of parameters you don't control.

MightyBot has more reason than most to care about this. Its agents do document-heavy work in lending and insurance, where a decision has to be explainable after the fact, and its write-up calls this "the hardest problem in agent observability: the same input can produce different outputs." Run the same document through the same agent twice and you can get two different answers.

The arithmetic here is worth noting. Suppose a failure happens one time in twenty. You rerun the request ten times to check, and it comes back right every time. That feels like evidence the failure has gone away. It isn't. With a one-in-twenty failure rate, there is roughly a 60% chance that ten reruns never show you the failure at all.

To be fair to my earlier self, my Triage article did say to run the test more than once, and to compare how consistent the raw model is with how consistent the app is. But it also said that a model which gets things right sometimes and wrong sometimes is "almost always application-layer". I would soften that now. The raw model has its own variance, so the comparison is between two failure rates, not between a consistent model and a flaky pipeline. And if you don't run it enough times, both rates look like zero.

What makes the question answerable is less about technique and more about what your team records. MightyBot's recommendations are sensible whether or not you buy their platform: record the complete input state for every run so a failure can be replayed exactly, fingerprint inputs so you can flag cases where identical inputs produced different outputs, and separate the deterministic steps from the model calls in your traces so you can see where the variance actually enters. Replay matters because it fixes everything except the model. Once the input, the retrieved context and the prompt version are held constant, whatever variance remains is the model's, and you can measure it.

The PM question that falls out of this is a simple one to ask an engineering lead: if a customer reports a bad answer, can we replay that exact request fifty times this afternoon? If the answer is no, every investigation starts from someone's memory of what probably happened.

Combinations, not causes

The bigger shift in my thinking came from something in Tian Pan's piece:

Most production AI incidents are a combination: a prompt with insufficient constraints (prompt failure) operating on inputs from a shifted distribution (data failure) that hits an edge case in the model's training (model failure).

Take any one of those three away and the incident doesn't happen. A tighter prompt would have held. Inputs from the old distribution would have been fine. A model without that particular gap would have coped. All three are causes. None of them is the cause.

That is a problem for a process built around finding a root cause, because the question "what was the root cause?" now has at least three true answers. Something has to pick one, and in practice I suspect it's usually whichever answer is easiest to point to first. The model is always available, because it's the one component nobody in the room owns.

It also explains why AI incidents are so slow to surface. Traditional software failures tend to announce themselves: an exception, a timeout, a stack trace you can follow backwards. Pan's observation is that an AI application instead "succeeds at producing the wrong output, confidently and at scale." There is no error to follow. And, as he puts it, "AI systems will produce incorrect outputs," which is not the incident but "the physics of probabilistic systems." The incident is that the wrong output got through, and that is almost always a question about several layers at once.

I made a version of this point in the Four-Layer Model, where I wrote that no single layer owns the blame. I think that was right but incomplete. Saying the blame is shared doesn't tell you what to do next. You still have to decide where the fix goes, and that decision is the part a root cause sentence will fail at.

We picked the wrong Toyota tool

The obvious way to rescue root cause analysis is to adapt the manufacturing categories behind the fishbone diagram for AI. I think the instinct is right, because the manufacturing lineage has a lesson in it that tends to get lost. (I wrote about Lean a while ago, and this is the part of it I didn't cover.)

There are two classic tools in that tradition, and they do different jobs.

The Five Whys came out of Toyota. Sakichi Toyoda developed it, and Taiichi Ohno made it part of the Toyota Production System. You ask why a problem happened, then why that happened, and keep going until you reach something you can fix. It is fast and memorable but it has well-documented weaknesses. The critiques collected on its Wikipedia page include a "tendency to isolate a single root cause, whereas each question could elicit many," results that "are not repeatable" because different people reach different causes, and an "inability to go beyond the investigator's current knowledge." Teruyuki Minoura, formerly a managing director at Toyota, called it "too basic a tool to analyze root causes at the depth necessary to ensure an issue is fixed."

The Ishikawa diagram, or fishbone, was popularised by Kaoru Ishikawa, who pioneered quality management at the Kawasaki shipyards and wrote it up in his 1968 Guide to Quality Control. Its whole purpose is to lay out the potential contributing factors to an effect, grouped into categories: people, machine, method, material, measurement, and in later versions, the environment. It doesn't pick one cause. It shows you all of them.

Read those Five Whys criticisms again with an AI incident in mind. A tool that isolates a single cause, on a failure made of several. A tool bounded by the investigator's knowledge, on a system whose most important component is a model nobody on the team can see inside. Ask "why?" five times about a wrong answer and the chain reliably stops at "because the model generated it," not because that's the deepest cause, but because it's the first one you can't open.

So I don't think the lesson is that manufacturing quality tools don't fit AI. It's that we reached for the single-cause one when the problem needed the one built for contributing factors.

Adapting the categories is what the fishbone was built for. As FlowFuse's guide to the Ishikawa diagram says, "pick categories that match your failure surface, and drop the ones that don't apply." For AI products, though, there is one mapping that is tempting and, I think, wrong in a way that matters for product teams: putting Material against training data. Most product teams don't control training data at all; it's a property of the model they chose, which makes it part of the machine. The raw material a product team actually works with is what goes into the model on each request. Here is how I'd map it:

Tian Pan's three-part incident fits across it cleanly. Prompt failure is people. Data failure is material and environment. Model failure is machine. And the reason nobody noticed until a customer did is almost always measurement, which is the bone most post-mortems forget to draw.

Which change, not which cause

If an incident has several causes, "what caused it?" stops being a useful question. The paper on Causal Software Engineering gave me a better one.

Its authors draw a line between two kinds of reasoning. "Correlational models answer 'what tends to co-occur'; they struggle to answer 'what would happen if we act.'" Associations in your data, they write, "aid detection and triage but do not justify actions." Their proposal is to treat changes as interventions: to ask what would have happened if you had done something different, and test it.

That academic framing turns into a very practical question after an AI incident. Not "what was the root cause?" but: which change, in which layer, would most cheaply and most durably have prevented this, and how would we know?

Some of this is easier to test than it sounds. Take the recorded request that failed. Rerun it fifty times as it was, and count the failures. Then change one thing at a time: fix the stale document, tighten the prompt, pass the same context to a stronger model, add an output check. Rerun each variant fifty times. You now have a failure rate for each intervention rather than an opinion about which one mattered. That's the reproduction test from Triage grown up: instead of asking whether the failure repeats, you're measuring how much each possible fix reduces it.

None of that is realistic by hand. One baseline and four changes at fifty runs each is 250 responses, and every one of them has to be judged right or wrong. Nobody is going to read 250 answers to the same policy question, and even if someone did it once, they wouldn't do it again next week. It has to be automated: something that reruns the recorded request, applies each change, and scores every response with a code check or a calibrated LLM judge. This is exactly the lesson software teams learned about regression testing years ago. Manual regression testing doesn't scale, so in practice it quietly stops happening, and automated suites won. I made the same point about evals in The Eval Gap: if running them "takes three days and requires booking someone's calendar, it won't happen." Triaging AI failures works the same way. If you haven't invested in automated test harnesses, you're going to struggle to triage AI issues at all, because the only option left is rerunning the request a handful of times by hand, and the arithmetic earlier shows how little that tells you.

Choosing between the interventions is a product decision, and I think it's one that belongs with the PM. The questions are the ones PMs already ask about any trade-off:

• Cost. A document refresh takes hours. A model switch can take weeks and change your unit economics.
• Reach. Does this fix one instance, or a whole class of failure? A stronger grounding instruction helps every policy question. Fixing one stale document helps one.
• Durability. Will this fix still hold after the next model upgrade?

That last question is the one I hadn't thought about properly until I heard Cat Wu on Lenny's Podcast. She described the Claude Code team auditing the product when a new model ships, and removing interventions that had been put in place to compensate for weaknesses in an older one. Her example was a to-do list feature, originally added to make sure Claude finished every call site in a large refactor, which became unnecessary once Opus 4 could handle that on its own.

The implication for incident fixes is that anything you add to compensate for a model weakness has a shelf life. It may become unnecessary. It may start to interfere with a model that no longer needs it. Either way, a fix placed in the prompt layer to patch a model limitation is a decision you will need to revisit, and you can only revisit it if someone wrote down which layer it went into and why.

So the "primary layer" of an incident, to my mind, isn't the one that caused it. It's the one where an intervention buys you the most.

What a post-mortem should leave behind

If the output of an AI post-mortem isn't a root cause, what is it?

Pan's answer is a set of concrete artefacts: a regression test case, an observability gap identified, an update to how you classify failures, and a guardrail or constraint added. I would add one more: the failure rate before and after the fix, measured by rerunning the request that failed. That's the evidence that the change you made is the change that mattered.

Each of those artefacts feeds something that already exists. The regression case goes into the golden dataset, which is the loop I argued in Eval Theatre has to close for any of this to work. The observability gap becomes a backlog item for the measurement layer. The classification update means the next incident of the same shape gets recognised faster. The guardrail is the intervention itself, recorded with the layer it lives in, so it can be revisited when the model changes.

A post-mortem that ends in a single root cause sentence and produces none of those is the incident version of eval theatre. It looks like a discipline, it is written up like a discipline, and nothing is different afterwards.

There's a simple test for whether a post-mortem has stopped too early. Read its conclusion and ask whether it would read the same for every AI incident you've ever had. "The model hallucinated." "LLMs are non-deterministic." "This is a known limitation of large language models." All true, and useless, because none of them tell you what to change. And if the honest conclusion is that the model can't do this reliably and you're choosing to live with it, that's fine, but write it up as a decision, with an owner and a date to revisit it, not as a cause.

The question I'd take into the next AI incident review is not "what was the root cause?" It's: if this happens again next month, which change will we wish we had made, and how will we know it worked?

Read on stvpj.com

You're receiving this because you subscribed at stvpj.com. Reply to this email anytime.