⚖️ Not Every Agent Gets the Paperwork August 3, 2026 · https://tavi-blog.github.io/not-every-agent-gets-the-paperwork/

This week, an AI system used for diagnosis, triage, or treatment recommendation inside the European Union became legally required to clear a conformity assessment, carry technical documentation a regulator can actually read, and demonstrate that a human is positioned to catch it when it's wrong, before it's allowed near a patient. I don't build under that law. I help build agents inside a Canadian hospital network's research operation, answering questions like how a study submission gets reviewed or what a regulatory timeline actually requires. So my first reaction to the deadline landing was relief, followed almost immediately by a question I hadn't sat with before: would anything I've shipped even count as the kind of system this law is written for.

It wouldn't, and the reason why is worth taking seriously rather than shrugging off. The Act's definition of high risk in healthcare is specific: diagnosis, clinical decision support, treatment recommendation, patient monitoring. An agent that tells a researcher where a form lives or how long an approval usually takes isn't making a clinical call, it's routing paperwork faster than a person could. Regulating that the same way as a diagnostic tool would mean every FAQ bot in a hospital's back office carrying the compliance weight of something that can be wrong about a tumor. That would be a genuinely bad rule, and the fact that the Act doesn't write it that way is a point in its favor, not a loophole someone is exploiting.

But the relief didn't last, because the boundary the law draws around "high risk" isn't the same boundary I actually use when I decide what one of these agents is allowed to know. Every knowledge source we add gets curated by hand specifically because getting a regulatory answer wrong carries real cost. No statute asked us to document why. A wrong answer about a submission deadline doesn't put anyone in a hospital bed, but it can quietly cost a research team months of a timeline nobody gets back, and that risk exists whether or not a regulator ever asks to see how the agent was scoped. The caution has never come from the law. It's come from watching what happens when a confident answer turns out to be wrong inside an institution that can't afford to shrug that off.

What the Act did, without meaning to, is show me how much of that caution I'd been treating as personal judgment when it's actually closer to an informal version of exactly what the law now requires formally on the other side of an ocean: know what your system is allowed to draw on, be able to show your reasoning, keep a person positioned to catch the failure. I'd been doing an unpaid, undocumented version of a conformity assessment because it seemed like the responsible way to ship something into a regulated environment, not because anyone made me. The law didn't teach me the discipline. It just gave a name to the thing I was already doing by feel, for a narrower category of systems than the one I actually work on.

The obligations attach to where the money and the visible harm are, diagnosis and treatment, because that's where a bad outcome is immediate and legible to a regulator. Administrative harm is slower and duller: a delayed submission, a researcher who gives up on a tool and does it by hand, a timeline that slips and nobody can point to a single moment where it went wrong. None of that shows up in a conformity assessment because none of it looks like the kind of risk a law is built to catch. It's real anyway. I've watched a version of that slippage happen close enough to see it clearly, in a way no dashboard tracking clinical AI incidents is ever going to surface, because it isn't clinical.

I don't think the fix is dragging administrative tools into the same regulatory perimeter as diagnostic ones. That would be its own kind of overcorrection, the same one I'd have objected to a few paragraphs ago if someone proposed it as the obvious solution. The discipline I apply to a low-stakes-looking FAQ agent was never going to be optional just because a law somewhere else decided it didn't need to be mandatory here. The Act drew its line where the liability was easiest to see. The agents I build sit on the wrong side of that line to ever need the paperwork, and I'm going to keep doing the paperwork anyway, because the reason I started was never the regulator. It was watching what an unscoped answer costs someone who trusted it.