🗝️ The Leak Was in the Plumbing, Not the Policy September 5, 2026 · https://tavi-blog.github.io/the-leak-was-in-the-plumbing-not-the-policy/

The examples are the part that stuck. A test result sent to the wrong patient. A medication record pulled into a message that was only supposed to confirm an appointment time. Those two lines showed up in a survey of several hundred healthcare leaders released this week, cited as illustrations of why a third of the AI agent rollbacks in health systems this year trace back to data leakage or exposure of information the agent should never have surfaced in the first place. The headline number is that three out of four healthcare organizations running an agent in production have had to pull one back or shut it down over a governance failure, and leakage is the single biggest reason inside that number.

The report's framing is that this is an oversight problem. It points to a gap between technical staff and business leadership inside the same organizations, where the people closer to the system report far more rollbacks than the executives sponsoring the program, and reads that gap as a communication failure, different levels of the org not seeing the same picture of how the rollout is actually going. That's a real finding and worth taking seriously on its own terms, because a program where the people funding it are more confident than the people running it is a program heading somewhere nobody voted for. I don't doubt that gap exists, and I don't doubt it matters for how these programs get resourced and reviewed.

But I've spent a long stretch of this year scoping exactly the kind of thing that produces those two examples, and neither one reads to me like a governance failure. They read like an architecture failure that happened to surface in a moment a compliance officer noticed.

A medication record showing up in an appointment confirmation traces back to a knowledge source or a data connection that got scoped too wide for what that particular message actually needed to answer. Every agent I've helped configure pulls from a defined set of sources, and the discipline is almost entirely about what you leave out. Give it access to more than the narrow slice of information the interaction requires and you haven't made it smarter, you've made it more likely to reach for something adjacent and technically true but completely wrong for the context it just got asked about. The fix for that isn't a new sign-off step in a steering committee. It's someone sitting down and auditing, source by source, what each agent can actually see, and cutting scope until the blast radius of a bad retrieval is small enough not to matter.

The wrong-patient result is the other failure mode, and it's the one I'd bet money on: an ambiguous join somewhere upstream, two records that share a lookup value they shouldn't, a key that resolves to more than one person when the schema assumed it wouldn't. I've chased that exact category of bug in dashboards that had nothing to do with AI at all, long before agents were part of the conversation, and the pattern is always the same. Nobody decided to build an unsafe join. Someone built a reasonable one against data that had a duplicate nobody had reason to expect, and it sat there working fine until the volume or the source changed enough to expose it. An agent sitting on top of that pipeline doesn't introduce the flaw. It just executes it faster and with more confidence than a person would have, and does it in a channel where the mistake reaches a patient directly instead of getting caught by someone reviewing a report before it goes out.

None of this means the governance framing in the report is wrong to want. Leadership genuinely does need a layer that tracks whether a program is healthy, and a perception gap between the people building the thing and the people funding it is a real problem worth fixing on its own. I'd just push back on the idea that fixing that gap fixes the leak. You can align every executive and technical lead in a health system perfectly and still ship an agent with a knowledge source three sources too wide, because closing that gap is an auditing cost nobody wants to pay for work that produces no visible feature. Scoping down access is invisible when it goes right and only shows up in a report when it goes wrong, which is exactly the kind of work that governance frameworks are good at requiring on paper and bad at actually funding in practice.

The report will get read as a case for more oversight, and it should be. I just don't think anyone rolling out the next agent is going to trace their two-line failure back to an audit that never happened, because that line never makes it into the postmortem. It gets filed as a policy gap instead, and the person who could have caught it moves on to the next dashboard.