One Concept, One Example, One Question

The concept

A successful test is useful, but it can tempt people into saying more than the test has proved.

I think of this as the edge of the evidence. A result tells you what happened under the conditions in which it was produced. Change those conditions and the claim may no longer hold, so the boundary around the result is part of the finding.

This is easy to lose when the work is under pressure. A prototype passes and confidence rises, but confidence is not the same as permission to make a bigger claim.

The more useful move is to ask what the evidence now allows you to say. Sometimes the answer is strong enough to say that this workflow can handle the normal case. Sometimes all you can say is that the approach works on a controlled sample. Before deployment, though, the team still needs to settle security and ownership; it also needs proper support and a live comparison. That narrower answer is still progress because it stops the next decision being built on implied proof.

Good evidence supports a claim and shows where the claim stops.

The example

Imagine a team testing a tool that helps review documents before they're shared.

In the first test, the tool identifies sensitive terms in a sample document and marks them clearly. An independent reviewer checks the resulting version and agrees that the sensitive material has been removed.

It would still be too much to say that the tool is ready for live confidential work.

The test has shown that one review route works on a known sample with a defined checking process. It hasn't shown that the tool is safe with messy real documents or that uploaded files are handled under the right retention rules. The team also lacks a route for challenging a missed finding, and ownership of a bad outcome remains unresolved. None of this invalidates the prototype; these are the next edges of the evidence.

The practical difference is in the next sentence the team is allowed to say.

They shouldn't say, "the tool works, so we can deploy it." A better claim would be: "the controlled sample passed, so the next test should use an agreed non-confidential corpus, include the full access route, and compare the result with the current manual process."

That sentence is less exciting, but it's much more useful because it turns the result into a decision about what to test next. It also protects the team from treating a technical success as though it had already answered how the work would operate or meet its legal obligations.

The same habit helps outside technology. A presentation narrative that makes sense to one internal audience may still confuse another; the evidence is real, but it has edges.

The question

Ask yourself:

What claim does this evidence allow us to make, and what claim would go beyond it?

If you can answer that cleanly, the next step usually becomes clearer. You either have enough evidence to act, or you have a more honest test to run.

Elsewhere this week…

• I published The Real Choice Behind Open and Closed AI, on why those labels are too blunt for deciding who retains control and where responsibility sits.

• The SecondBrain Framework now includes synthetic behavioural tests for Daily Note authority and other protected workspace boundaries. Its recurring-task guidance better separates unfinished user work from automated events, and framework validation now runs on pushes to main.

Support

Everything I publish here is free to read, and always will be.

I maintain this newsletter and my website in my own time. If something here has been useful to you, and you'd like to help cover the costs behind that work, you can support me through Ko-fi.

Generative AI is a powerful tool. I openly use it to conduct research, challenge my thinking, generate drafts, and review my work. AI never owns what I think, or my personal judgement. Everything I publish is a reflection of my own thoughts and opinions.