Hi,

Last month's update was about giving agents tools. This one is about proving they work, because I found out the hard way that those are not the same thing.

My own checklist passed all eight tools

I wrote a tool-design skill distilled from how I build agent tools, then pointed it at Cameron's eight hand-written tools. They were built against those exact principles. The audit found four contract bugs anyway.

The one that stings: Cameron reported $2,812 of dining against a real figure of $2,752. The tool returned 50 rows out of 262 and said count: 50. Perfectly true, and read as a complete total.

Design review cannot catch these, because every one of them passes it. The full audit is in Testing Tools for AI Agents.

Cameron v2 ships with evidence instead of assurances

Cameron hit v1 by getting tools that touch your money. v2 builds the layer that proves they behave: unit tests over what tools return, plus an eval harness that drives the real agent against a sandbox Postgres and grades what it actually did.

Eleven cases, grouped by what can go wrong rather than by feature: tool mis-selection in both directions, a capped page reported as a complete total, the approval gate, prompt contracts, CSV import. Graded by plain deterministic functions, no LLM judge, because the answer is usually a number and a number has one spelling.

The evals immediately found a new bug. run_sql returned SQL NULL for a category that does not exist, a payload byte-identical to a category that exists with zero spending. The agent guessed. Same defect class as v1's truncation bug: a payload the agent predictably misreads. Fixed in the same release.

Cameron also got its own look, and the series hub now shows it running rather than describing it. Watch the approval gate do its job on a CSV import, showing the exact arguments before anything is written, at blog.agentailor.com/cameron. Devlog 3 covers where the first cases came from: How to Write Your First AI Agent Evals.

If you read one thing

Building an Agent Eval Harness. A harness is four components: the case, the runner, the capture, and the graders. The guide works through what each one has to do, built around two real harnesses, so you can decide whether to build one, adopt a framework, or combine both. There are hundreds of eval tools now and a coding agent will draft you a harness in an afternoon, which makes that choice harder, not easier.

Its companion, How to Write AI Agent Evals That Prove You Wrong, is about the cases themselves: sourcing them from real traces, spotting the ones that pass for the wrong reason, and when to overrule a judge.

The blog agent can now brief your coding agent

Type /handoff in the chat widget on the blog and it packages the session as a build spec for whatever coding agent you use. Not a transcript. The decisions the conversation actually reached, pointers to the articles instead of paraphrases of them, and an explicit list of what was never verified.

It came out of watching what people actually use the agent for. They are not looking up facts. They are working through architecture and implementation details, which auth model to use, where to host it, what not to put in a tool handler. By the end the conversation holds a real spec that exists nowhere else, and then the tab closes and it is gone.

Two more worth your time

Agent Plugins Explained. An open standard for packaging Skills and MCP servers into one portable folder. Vercel proposed it, most of the large vendors refined it, Google joined as a core maintainer the day it published. This one is ecosystem news, not my news.

Your System Prompt Has a Shelf Life. Anthropic deleted over 80% of Claude Code's system prompt for Claude 5 and measured no loss. Those instructions did not become wrong. The model outgrew needing them. Yours are aging the same way.

Two new skills are up at agentailor.com/skills, including the eval-case skill behind Cameron's suite.

— Ali

If you'd rather not get these, unsubscribe below. No hard feelings.