🩺 Someone Still Has to Watch October 11, 2026 · https://tavi-blog.github.io/someone-still-has-to-watch/

The detail that caught me in this week's Lancet writeup on Google's AMIE chatbot wasn't the diagnostic accuracy. It was the four criteria. Before the trial started, the supervising physicians were trained on exactly four reasons to interrupt a patient's conversation with the AI: a risk of harm to the patient or someone else, real emotional distress caused by the interaction itself, a clinical concern the supervisor spotted on their own, or the patient just asking for a person. Across ninety-eight completed conversations, nobody needed any of the four. That's the headline. Zero interventions.

I understand why that's the number everyone is leading with. A chatbot talking to patients before a visit and triggering no safety stops is a genuinely good result, better than a lot of people, including me, would have guessed going in. If the bar is "does this thing do harm when nobody's watching," it cleared the bar. The physicians who reviewed the transcripts afterward said it helped them prepare for the visit three times out of four. That's not nothing. I don't want to undersell a result just because the framing around it bugs me.

What bugs me is the phrase "nobody's watching." Somebody was watching the entire time. Every single one of those ninety-eight conversations had a licensed physician monitoring it live, trained specifically on when to step in. The zero isn't a measurement of an unsupervised system behaving itself. It's a measurement of a supervised system behaving itself while the supervision cost doesn't appear anywhere in the number being reported.

I think about this because I've built a model that had the same problem, at a much smaller scale and with much lower stakes. A few years ago I put together a predictive model for how long a clinical study submission would take to get through initial authorization, mostly because the variance was driving everyone doing resource planning a little crazy. The model itself, the actual machine learning part, came together fast. What took real effort was getting the prediction in front of the people who needed it without requiring them to run anything on their own machine. Nobody on that team had Python installed, and nobody was going to install it for this. The entire second half of the project was building a path from a trained model to a number someone could see in a spreadsheet they already had open.

That's a deployment problem, and it's the kind AI coverage tends to skip past, because it's boring compared to the model. But the AMIE study has a deployment problem that's worse than mine, because mine was solved once and then it stayed solved. Once the scoring pipeline worked, it worked for every new submission at no additional cost. The AMIE study's solution doesn't scale that way at all. The thing that made the zero possible was a person's attention, and attention doesn't get cheaper as you add more patients. If a clinic wants to run this across a full patient load instead of a single feasibility study, it needs either a lot more physicians sitting through transcripts in real time, or it needs to loosen the supervision that produced the zero in the first place. Neither option is a software update.

I've watched a smaller version of this inside the AI agents I've helped build for a research team. The part that eats time isn't writing the agent. It's deciding what it's allowed to know, because every source you hand it is also a new way for it to be wrong about something regulatory, and someone has to keep checking that. That review work is exactly as unglamorous and exactly as uncountable in a press release as a physician watching a transcript, and it's also exactly where a promising pilot quietly stalls before it ever reaches the rest of the building.

So when the next version of this study gets published, with more clinics and more patients, I already know which number I'll be looking for first, and it won't be the diagnostic accuracy. It'll be whether the supervision held the same shape, or whether it quietly thinned out to make the bigger number possible.