🚧 The Checkpoint Is Watching the Wrong Hands October 18, 2026 · https://tavi-blog.github.io/the-checkpoint-is-watching-the-wrong-hands/

The line in the FDA's discussion paper on generative AI medical devices that stopped me wasn't about hallucination rates or benchmark scores. It was the description of what an agentic system needs before anyone should trust it: a human-oversight checkpoint, placed before the system takes any high-consequence action. The public comment window on that paper closes today, and I've been turning the sentence over since I read it, because the agents I help scope inside a hospital network's research operation already live under something like that checkpoint, and it sits in a place the sentence doesn't seem to expect.

For a system that actually acts, the checkpoint makes complete sense, and I don't want to undersell how reasonable it is. If an agent can adjust a dosage, schedule a procedure, or send something to a patient on its own, there's a real moment where its hand is about to move, and a person needs standing to grab the wrist before it does. That's not bureaucratic caution for its own sake. It's the only design that makes autonomy survivable in a setting where a wrong move has an immediate, physical cost. I'd want that checkpoint built the same way if I were the one shipping a system with that kind of reach.

What the paper treats as the same category, under its own agentic-systems section, is a much larger and quieter set of tools that never act in that sense at all. The systems I've helped build answer questions. They tell a researcher how a submission review usually proceeds, or what a given regulatory timeline requires, and then a person reads that answer and goes and does something with it, somewhere else, often on paper, often days later. There's no moment where the agent's hand is about to move, because the agent never has a hand in the first place. The action belongs entirely to the human who read the output and believed it. A checkpoint built to catch a system before it acts has nothing to grab onto here, because the thing worth catching isn't an action the system takes. It's a belief the system produces in someone else, which then turns into an action outside the system's view, in a different form, on a different day, filed by someone who may not even remember which part of it came from the tool and which came from a colleague down the hall.

That gap matters more than it looks like it should, because it's exactly the shape of risk that's easiest to scope conservatively and easiest to describe badly afterward. I spend real effort narrowing what one of these systems is allowed to draw on, specifically because I know the failure mode isn't the agent doing something harmful. It's the agent sounding confident about something it shouldn't have answered, and a person downstream treating that confidence as settled. A framework built around intercepting actions before they happen has no natural place to put that failure, because by the time anyone could point to the harm, the system is three steps removed from it and the record only shows a person who made a reasonable-looking call.

I don't think the paper is wrong to lead with action-taking systems. That's where the sharpest, most legible version of the risk lives, and regulators reasonably start where the harm is easiest to trace. But the comment period closing today is asking, among other things, how broadly the word agentic should stretch, and an advisory tool that only ever produces text gets swept into that word just as easily as something that can actually move a dosage. Lump them together under one checkpoint model and you get strong protection for the rarer case and something closer to a shrug for the one I actually work inside of, where the harm is real but shows up a step removed, in someone else's decision, in a form nobody designed a checkpoint to watch.

I don't know what fraction of today's submitted comments will raise that distinction instead of arguing about benchmarks and liability. I'd guess not many, because the people with the clearest view of it are the ones scoping advisory systems from inside institutions too busy running them to write in. I'm not writing one in either. I'm just noticing that the checkpoint everyone's debating assumes a hand that, in my part of this work, was never there to grab.