One Concept, One Concept, One Question

The Concept

A workflow can look sensible until people try to use it. A team designs a process, tests it in a clean setting, and then discovers that the real work depends on things the test did not include.

It may be useful to separate two checks: 1. Ask whether the idea makes sense. 2. Ask whether people can use it in practice.

A clean test is easier to arrange, and it usually gives people a result they can feel good about. It shows that the concept is plausible, the demo is smooth, and the intended benefit is visible.

The harder test includes permission problems, handoffs, and unclear decisions. It checks whether the process works when people use it to do a real task.

The Example

Imagine a team building an internal assistant to help staff answer common policy questions. This is a constructed example, but it is deliberately ordinary.

In a demo, it works well. Someone asks a question, the assistant finds the relevant policy, and the answer is clear enough to check. The group can see why the idea is attractive: it could reduce repeated questions and help staff get to the right source more quickly.

A member of staff opens it from the place they normally start work. They use their ordinary account. They ask a question from a live support conversation, not a prepared example. The assistant points to a policy, but the route also has to handle the work around the answer: whether the source is current, whether the person can see the underlying material, and whether there is somewhere to go when the answer is not enough.

If the person has no clear next step, the assistant has not really helped them finish the work. It has moved the uncertainty from a visible conversation into a tool output. The same problem appears when nobody owns the policy correction, because the tool can surface the stale answer without creating a route to fix the source.

That kind of test may make the first result look less impressive because it reveals the work around the answer. The assistant may need a narrower scope, clearer source ownership, or a proper escalation route. But that is not failure so much as useful evidence. The team has learned what would have to be true for the assistant to be used responsibly in normal work, instead of learning only that the model can answer a prepared question.

The Question

Ask yourself:

Have we tested the idea, or have we tested the route people will actually use?

If the answer is only the idea, the next useful step is probably not a bigger launch. It is one carefully chosen end-to-end trial that includes the conditions that will decide whether the work can hold up outside the room.

Elsewhere this week...

• I published Friction Is Part of the Interface, which argues that good AI systems should remove wasteful effort while preserving the friction that keeps human attention, learning, challenge, and judgement engaged.

Support

Everything I publish here is free to read, and always will be.

I maintain this newsletter and my website in my own time. If something here has been useful to you, and you would like to help cover the costs behind that work, you can support me through Ko-fi.

Generative AI is a powerful tool. I openly use it to conduct research, challenge my thinking, generate drafts, and review my work. AI never owns what I think, or my personal judgement. Everything I publish is a reflection of my own thoughts and opinions.