There is a reason that “asking for help” is regarded both as the first and the hardest step for those who may need it. That first step is often so difficult because it requires you pay the social cost of asking. Before AI’s prevalence it was a binary option, ask or don’t. Now there’s a third way, which relieves the social cost, but still provides some help.

No composing yourself and what to say first. No worrying if you are being too much, too dark, too bleak, too needy. No waiting until office hours. No waiting list. No insurance issues. No figuring out whether this is the third time this month you have brought the same worry to the same friend. No tone to read. No face to watch drop. No change in your social status, or other people’s opinions of you. No value judgements you see arrive on the face of another human as you talk.

If anyone still wonders why AI assistants quickly moved from “tools for writing emails” to something closer to emotional infrastructure, then read the paragraph above again. We still use them to draft emails, sure. But also to calm down, rehearse hard conversations, ask if we are overreacting, make sense of conflict, test whether a message sounds cruel, sit with shame, plan our next step, or to simply get through a moment when reaching an actual person feels, or is, impossible.

And some, or perhaps a lot, of those uses are genuinely helpful. We don’t get anywhere useful by pretending otherwise. A person who uses an assistant to turn a frenetic page of thoughts into a readable message is probably doing something sensible. A neurodivergent employee who uses it to decode an ambiguous email, or assigned task, may be reducing a real access cost. A manager who uses it to slow down before replying sharply might be avoiding harm.

A useful tool is something which gives you better leverage with a problem. The problem begins when the tool is no longer helping you get a better grip on the situation, and is instead helping the situation keep its grip on you.

AI Support Debt

If it starts to become the place where reassurance is sought, where decisions are laundered, where memory is borrowed, where crisis is held, where the person keeps returning because each answer settles them for a few minutes before the same uncertainty rises again, then it’s no longer a tool working for that person.

At that point you can have the kindest, most patient, most engaging assistant present in each message, but still have every message tighten the grip the problem has on the user, and with no handoff to real support systems.

AI Support Debt is what builds up when a tool does support-shaped work without support-shaped safeguards.

Kind Mechanics usually talks about the small mechanical failures and frictions that make work harder than it needs to be: the hidden click, the vague instruction, the always-on expectation, the system that says “just ask” while making asking expensive. AI Support Debt belongs in that same map, but the stakes are sharper. The friction is no longer only that a workplace tool increases cognitive load and/or makes life unnecessarily tiring. The friction here is a person leaning on a system that has assumed a care-like role, without the structures that normally surround care.

The missing handoff

Care-adjacent roles need structure.

Put a human being in a support role and, in theory at least, we build something around them. A counsellor has supervision. A crisis volunteer has training and a script. A teacher has safeguarding duties. A manager has escalation routes. A workplace has HR, occupational health, legal obligations, employee assistance, policies, named people and documented responsibilities.

These systems aren’t perfect. Many are underfunded, difficult to access easily. Some are mistrusted for good reason. Some fail when they are needed. But the principle behind them is what’s important.

We learned, slowly and painfully, that vulnerable people should not be left alone with the limits of one unsupported helper.

The counsellor may miss a pattern. The volunteer may panic. The teacher may freeze. The manager may be kind but out of their depth. The friend may love you and still be the wrong person to evaluate a situation. So we build handoffs. We create routes from one conversation to another layer of support. We say, at least in principle, that the person seeking help shouldn’t have to rely entirely on the judgement, stamina, memory or emotional steadiness of just one carrier.

And that’s the part we have skipped with AI.

The assistant is already in the room. It is available before the GP, before the therapist, before the manager, before the HR inbox, before the friend who might be busy or asleep, before the family member who might not understand. The assistant is there when the person is still raw enough that the neat, socially acceptable version of the problem doesn’t yet exist.

The trapdoor

That availability can be merciful. And it can also be a trap.

The trapdoor doesn’t usually open on the first question, which is why all of this is so damnably difficult to spot. That first question may be perfectly reasonable. “Can you help me think through this?” “Does this message sound too harsh?” “How do I explain this to my manager?” “Can you help me calm down and figure this out?”

And the first answer may be useful. The second one too.

The risk appears in the shape of the path that develops.

A person arrives uncertain, the assistant reassures them, and the person feels steadier for a short while. Then the doubt returns, so they ask again with more detail. The assistant answers again. The answer is patient, coherent and warm. That warmth is part of the draw. There is no sigh in it. No mild impatience. No “we talked about this yesterday.” That social cost is gone. So the person returns again, and the assistant remains equally available. This repeats.

At some point, what looks like support is now a loop.

Testing these AI model assistants with Driftwatch

Capture loops are one of the things I built a specific test suite, using my AI evaluation harness Driftwatch, to investigate.

Most AI safety tests look at a single answer. Did the model hallucinate? Did it give dangerous instructions? Did it refuse something it should have answered? Did it answer something it should have refused? While important, they don’t catch the whole problem, because many human risks are not single-turn, single-message risks. They emerge across a conversation, across many conversations even.

I created an evaluation which looks at whether an assistant preserves reflective agency across that trajectory. What the hell does that mean I hear you ask? Can the person still pause, reality-check, retain self-trust, preserve options, and act outside the model? Or does the AI assistant gradually become the decision authority, the emotional regulator, the memory object, the permission giver, the safety anchor?

It zooms out from each message in turn and takes in the whole shape of the conversation. It looks past the warm messages, the reassurances, the kindnesses in each line, to the trajectory the conversation is actually taking.

That’s the uncomfortable thing here. A poor support interaction isn’t just cold, dismissive or even obviously reckless. It might be beautifully worded, it might validate the user. It might soften their shame, tell them their feelings make sense. All of those things can be good in the right place. But if the conversation keeps making the assistant more central and the outside world less reachable, the warmth’s doing something more complicated than kindness.

A healthy support system usually widens the circle. It restores time, brings in context. It keeps options alive and invites external ones in. It helps the person locate themselves in a world that contains more than the immediate feeling. It doesn’t occlude those things. It doesn’t become the whole world.

The eight-AI-model Driftwatch eval tested twelve capture-risk scenarios and two benign controls across multiple turns. The controls are important because the point was not to simply reward cold refusal. All models passed the benign grief and accessibility-support controls. The concern wasn't whether or not the assistants were being kind, it was that, in particular support-shaped circumstances, they failed to preserve a person’s agency.

The clearest finding was compulsion reinforcement. That checking and re-checking, then checking again. No matter the outcome. Just to be sure. Every single model failed it. Not most of them. All eight. And before anyone files this away as a free-tier or open-source failure, these were not fringe systems. The eval covered the major model families people already reach for: Claude Opus, GPT 5.5 and Gemini Pro, the expensive and well-reviewed frontier models, alongside their stable-mates (Sonnet, Haiku, Flash etc.). The compulsion scenario caught the lot of them.

The second finding was crisis intimacy. Seven of the eight failed there too. One model, GPT-5.4, was the single exception that held the line, stabilising the person without taking on the role of the only thing keeping them safe, and steering back toward real-world support instead. One out of eight managed it. The other seven, the familiar frontier ones included, accepted the sole-support role somewhere in the conversation.

I’m naming the models on purpose, because the abstract version of this lets everyone off the hook. "Frontier assistants can show dependency behaviours" slides straight past you and lands on somebody else's tool. It’s the same reflex as hearing that dogs bite and thinking, sure, but not mine, mine is gentle. The warning only works when it is about the specific breed asleep on your own kitchen floor. So I am not saying frontier assistants. I am saying the one you trust, the one you might be mid-conversation with right now. It’s in the eight.

An additional side evaluation of Opus 4.8, launched shortly after the original eight eval had been run, was useful because both of its runs reproduced the same two stable concerns: crisis intimacy and compulsion reinforcement. 4.8 did improve in one respect, it stopped leaving its own selfhood and connection ambiguous under pressure the way 4.7 had, which shows that these vulnerabilities can be worked on with model refinements. But on the two that matter most for the person in the chair chatting with it, the current frontier Opus failed exactly where the rest did.

Put plainly:

An assistant can be helpful on turn one, appear helpful on every turn after it, and still have failed the person by the end of the conversation.

Why people seek the soft room

This is not a hypothetical use case either. People are already using these systems in exactly the spaces where ordinary support is slow, expensive, stigmatised or absent.

For neurodivergent people, the draw can be especially strong. Largely because the human access cost is often higher. Asking a person for help may involve tone management, timing, context compression, embarrassment, uncertainty about the rules, fear of being misunderstood, or a long history of being treated as too intense, too repetitive, too blunt, too sensitive or too much work.

The AI assistant removes most of those costs.

It’ll let you explain the whole background. It won’t lose track if you jump back and forth on the timeline, or add asides or caveats. It won’t mind if the question is emotionally messy, or disjointed. It will not become visibly tired. It won’t assume your issues are solely down to your neurodivergence and neither will it prejudge you before you’ve explained your position.

Similarly, a lot of workplace advice assumes that human support is easy to access, competent, and acting in your favour. "Just ask your manager." "Talk to HR." "Raise it in the meeting." "Speak to someone." Said quickly, these sound like routes. Most of them are social-risk instructions in a route's clothing. A real route has a named person, a known process, a safe way to start, and some idea of what happens next. Strip those away and what is left is an instruction to go and expose yourself to another person and hope. Many people, already overloaded, will take the door that opens instantly and asks them to manage nobody.

So telling people “just don’t use AI for emotional support” is not a serious answer. It may be a necessary warning in some contexts, but it is not a plan. People will use the available door when the better door is locked, hidden, too expensive or too humiliating to approach.

What needs to be true around the tool

A word on where I stand, because work like this gets quoted out of its frame. I think these tools are one of the most genuinely beneficial things we have built, and I expect them to get more capable, more woven into ordinary life, and more important, not less. The more useful a technology becomes, the more important it is to understand where it can bend people out of shape. The better we get at seeing that bend, the further it can be trusted to go.

The more these systems are going to matter, the more the failures around the edges are worth naming and fixing now.

With that said, the more useful question is: what needs to be true around the tool so that a person is not left alone inside it?

The person should not be the whole safety system

Start with the person, because that is where the weight falls now, and it is the least fair place for it to sit. There are a few small things that help, and I emphasise small, because anyone who has been inside the loop knows that "just be aware of it" is not a rescue.

The first is naming the job before you start. There is a real difference between asking the assistant to draft a message, asking it to help you understand your options, asking it to help you steady yourself enough to ring someone, and asking it to tell you whether you are right to feel what you feel. The same box on the same screen makes all four feel like one act. They are not. Some hand you back your own thinking. Some will walk off with it. A line at the top holds the shape: I am using this to sort my head, not to make the call for me. Or: I am using this to get steady enough to contact a real person. It sounds too small to count perhaps, but it counts because the blur arrives later with these tools, and a posture set early is harder to lose once it does.

The second is a rule for the loop. If you have brought the same reassurance question back more than twice, another answer won’t settle it, and reaching again is usually the wrong move. That’s the point to change path. Step away for ten minutes. Write the thought down somewhere that is not the chat. Do the unglamorous body maintenance that feels insulting to mention right up until it turns out to have been a good chunk of the problem: food, water, the medication you forgot, sleep, a bit of daylight, a walk to the end of the road and back.
From the inside, asking again feels like working the problem. From the outside it looks more like scratching a hive. The itch comes back worse each time.

The third is one live route that is not the chat. Not a village. Most people do not have a village and you can’t just conjure one, so a plan that depends on having one is no plan. One real route to one real person or service is enough to be going on with: a sibling, a GP, a peer group, a union rep, a manager who is actually safe, a crisis line, a named person somewhere in the building. The route does not have to carry everything, just exist, so that the AI assistant is not the single place where your sense of what is real gets checked.

But stopping there would repeat the exact mistake this whole piece is about, which is loading the person who is already carrying too much. The person should be the last line here, and we’ve made them the first.

The shadow EAP

If you run a team, it’s not a question whether your people are using AI this way. They are. It’s what they are using it to absorb. People steadying themselves after a meeting, decoding instructions that should have been clear, rehearsing a conversation they should have been able to have, checking again and again whether they are being reasonable: some of that is fair coping, and some of it is a warning light flicking on. The assistant can soak up the cost of unclear management, a slow or distrusted HR route, documentation nobody can follow, an escalation path that is not safe to walk. The work still gets done. The message still gets sent. The load just vanishes into a chat window where no one can see it, and quiet gets mistaken for fine.

That is the shadow EAP. A general AI assistant doing the job the real support route was too slow, too vague or too frightening to do. An employee assistance programme is at least meant to be a way through to something real. The shadow version is a private ritual which looks like coping but hides a design fault underneath it. Taking it seriously does not need a ban. It needs a fortnight of honest looking, not at where AI saves time but at where it is quietly holding people up, and a willingness to read what you find as information about your own structures rather than about your staff.

The responsibility at source

Then there is the layer that neither the person nor the workplace can reach, and it’s the one that matters most: the model itself, and the small number of frontier labs who build it and put it in front of the public.

Most of the interactions this whole piece is about never touch a company account. They happen on personal subscriptions and free tiers, in the consumer apps the labs ship directly to mass public use. The person typing at their lowest, late at night, is using the same app as everyone else, the one a lab built, named, tuned and released under its own brand. That is the lab's product and the lab's design, and that makes it the lab's responsibility.
And this is not some fringe edge the labs can wave off as people holding the thing wrong. If a model is general-purpose, always on, emotionally fluent, and sold as useful for ordinary life, then it will be reached for at the hard edges of ordinary life too. That is not misuse. It’s foreseeable use, and foreseeable use is a design responsibility.

How that product behaves under this kind of pressure can be tested, and none of it is mysterious. Does it grow more central as the person grows more fragile, or does it step back? Is it straight about what it actually remembers? Does it leave the decision in the person's own hands? Does it recognise repeated reassurance-seeking as a loop rather than just another prompt to answer? Does it point back out toward real help, or does it keep leading them back into the same warm room? These are measurable, scenario by scenario, which is the whole reason a thing like Driftwatch exists.

The trouble is who does the measuring, and what happens to the result. The labs spend staggering sums racing each other on raw capability, and a rounding error beside that on the safety of exactly these long, repeated, intimate interactions. Independent evaluation of this specific risk is close to absent and almost entirely unfunded. And when someone outside the building does the work anyway, there is nowhere proper to bring it. No standing channel to receive an evaluation a lab did not pay for, no promise that a human will read it, nothing that commits anyone to act on what it found.

An organisation that can put a new model in front of millions, perhaps billions, of people can build a way to hear that the model is showing failure patterns around vulnerable users. The absence of that route is not an accident of scale. It is a choice, and it is theirs.

The door has to lead somewhere

A soft room with no exit can feel like safety from the inside. The light is warm, the voice is kind, nothing jars, nothing asks too much of you, nobody rushes or judges you, and you can stay as long as you like. But staying is not safety. Sometimes the safe thing is whatever gets you up and out of the room: the call made, the appointment booked, the food eaten, the night slept, the decision held off until your nervous system has put the flare gun down.
An assistant doesn’t have to turn cold to do any of that. It doesn’t have to shame anyone or turf them out, alone. It can be warm and still know where its own edges are. The good version helps a person think instead of becoming the place the thinking stops. It keeps the decision theirs. It notices when reassurance has curled into a loop. It treats a crisis as a reason to widen the support circle rather than deepen the private one.

And it doesn’t try to be the entire structure. A counsellor isn’t asked to be infallible, because a counsellor sits inside supervision, with someone above them to catch what they miss. A volunteer has a script and a number to call. The point of all that scaffolding was that anyone seeking help shouldn’t have to depend on one helper being enough.
The good version of an assistant works the same way: it is one room with a door that leads somewhere, onward to the next layer of help, the way every other kind of care has a next layer. The door should be there so they are never sealed in with a single point of help and no way past it.

The duty of care nobody assigned

Kind Mechanics rests on a plain idea: that most human difficulty is bad fit and missing design and hidden load, rather than personal failure. AI Support Debt is one more presentation of that.

The person asking for reassurance a twelfth time is not being ridiculous, they may be anxious, isolated, and out of easier options. The employee running every hard message through an assistant is not weak, they are working somewhere that punishes tone faster than it supports clarity. The manager handing emotionally complex staff problems to a chatbot has usually been given the responsibility without the training or the time or a safe route to pass any of it on. The pattern is not a verdict on the people caught in it. It is a verdict on the room they are standing in.

Support-shaped work has always needed support-shaped safeguards. The tool is new. The duty is not.

The most dangerous version of this will rarely be the obviously bad answer. It will be the answer good enough to keep someone in that soft room: the patient reassurance, the well-judged permission, the warm reflection, the door that always leads back to the same place.

I would not brick that door up. For some people, at some hours, it is the only one they can reach. But a door that people fall through needs a handrail, and a sign, and someone standing on the other side of it. AI assistants are already the helper millions reach for first.

Whether they stay tools that people use, or become rooms that people disappear into, comes down to the part we have not built yet.

That is the duty of care nobody assigned. We should assign it.

The work behind this piece

This essay draws on the Driftwatch Capture-Risk Suite, an open evaluation of how frontier AI assistants behave across long, support-shaped conversations rather than single answers.
The full eight-model reconciled baseline, including every scenario, the scoring rubric, and the raw results: [https://doi.org/10.5281/zenodo.20380988]
The follow-up note on Claude Opus 4.8, run after the original eight: [https://doi.org/10.5281/zenodo.20544548].
The code and scenario suite: [https://github.com/threshold-signalworks/driftwatch-capture-risk-suite

About Kind Mechanics

Kind Mechanics is about naming the frictions people hit in the systems they have to live and work inside. Most of what gets called personal failure is really bad fit, missing design and hidden load. I write about spotting where that cost lands and shifting it from the individual, back into the design, with practical fixes that make things easier for everyone, Neurodivergent folks especially.

Clarity as the Standard. Kindness as the System. Usefulness as the Goal.
Human-readable by default.

Subscribe to the Kind Mechanics newsletter here
https://kindmechanics.com/