🩹 Caution Has a Side Door July 24, 2026 · https://tavi-blog.github.io/caution-has-a-side-door/

Somewhere in a hospital that isn't the one I work for, someone three patients behind is pasting a treatment plan into a free web tool because it will turn a paragraph of clinical shorthand into something she can hand off in the next ninety seconds, and the traffic leaving her browser during that exchange looks, to whatever security software is watching the network, exactly like every other browser session on the floor. Nothing fires. Nobody signed a data agreement covering where that text is going or what happens to it once it lands somewhere. A recent run of healthcare cybersecurity coverage has taken to calling this shadow AI, borrowing the term from a decade of shadow IT before it, and the framing is almost always the same: staff are quietly breaking the rules, and the fix is better detection, stricter policy, more training on what counts as a violation.

That framing deserves to be taken seriously before I push back on it, because the underlying risk is real and not exaggerated. A record that leaves an institution through a free tool with no business associate agreement behind it is a genuine exposure, not a paperwork technicality. It can end up stored, and in plenty of consumer-facing tools, used to improve the product for the next person who logs in. Traditional monitoring can't catch it because nothing about the request looks wrong; it's just a browser talking to a legitimate URL. Anyone who has spent time thinking about where regulated data is allowed to travel should find that unsettling, and I do.

But I don't think the story is mainly a rules problem, and I say that from a strange angle: I help build the sanctioned version of exactly this kind of tool inside a large hospital network's research operation. Part of what that work involves is designing agents that answer routine questions for people buried in submission timelines and regulatory paperwork, and the discipline in it is almost entirely about restraint. Every knowledge source an agent can draw from gets added deliberately, because a wrong answer about a regulatory deadline is a much more expensive mistake in this environment than in almost any consumer product, and a hallucinated answer that sounds confident is worse than no answer at all. So the agent that actually ships ends up narrow. It answers the ten questions it was carefully built to answer, and for everything just outside that boundary, it says nothing, correctly, because saying nothing was the safer design choice.

That narrowness is the right call, and I'd make it again. But it isn't free, and the bill doesn't come due where the decision got made. It comes due for whoever hits the eleventh question, the one the sanctioned tool was never scoped to touch, at the exact moment they don't have ninety seconds to route it through the slower, more careful channel that would actually handle it safely. The caution that makes an institutional tool trustworthy is the same caution that leaves a gap next to it, and somebody standing in that gap with a deadline is going to reach for whatever answers the question fastest, sanctioned or not. I don't think that person is careless. I think they're doing the same cost-benefit math anyone does when the safe option is also the slow one and the unsafe option is sitting open in another tab.

What the shadow AI coverage keeps missing is that this isn't a story about staff who don't understand the rules. Most of them understand the rules better than the reporting gives them credit for, and use the workaround anyway, because understanding a rule and being able to afford following it in the moment are two different things. A policy telling someone not to paste a chart into a free summarizer doesn't change what happens when the sanctioned system can't answer her question in the time she has to answer it. It just makes the workaround a violation instead of a workaround, which helps an audit and does nothing for the person still three patients behind.

I don't think the fix is a faster policy rollout or a longer list of banned tools, because neither one touches the actual gap. The gap only closes if the sanctioned side gets fast enough, or broad enough, that reaching past it stops being the rational move. Until then, every carefully scoped agent I help narrow down to something trustworthy is also, quietly, defining the exact shape of the door someone else is going to walk through instead.