Eval Theatre
Eval Theatre
I spent a lot of time this summer reading about evals. Some of that was deliberate: I wrote about evaluation last year as the biggest genuine gap most product managers have, and I wanted to know whether that was still true or whether the field had moved on while I was looking elsewhere. Roughly what I found was:
- Why AI evals are the hottest new skill for product builders - Lenny Rachitsky with Hamel Husain and Shreya Shankar, which has become something close to the canonical text on this
- Why most AI products fail - Lenny again, with Aishwarya Naresh Reganti and Kiriti Badam, who between them have shipped more than fifty enterprise AI products and are noticeably less enthusiastic
- Evals Are NOT All You Need - the same pair writing in O'Reilly Radar, and the piece that made the argument stick for me
- How I Designed and Implemented Evals for Product Talk's Interview Coach - Teresa Torres, and the best public worked example I have found
- LLM Evals: Everything You Need to Know and "It's Hard to Eval" Is a Product Smell - Hamel Husain
- Can public chat data predict real-world AI misalignments? - OpenAI's alignment team, on why evidence from real usage beats evidence from tests
- Eval awareness in Claude Opus 4.6's BrowseComp performance and A global workspace in language models - two Anthropic reports that got genuinely strange, in ways I will come back to
- AI enthusiasts are in a race against time - Charity Majors, on AI as a magnifier of whatever discipline an organisation already has
- and a number of eval tooling comparisons, each written by one of the vendors being compared, and each concluding that the vendor who wrote it came out best
What I expected to find was a maturing practice. Better tooling, clearer patterns, some consensus forming about what good looks like. What I actually found was an argument about whether any of it works.
That surprised me more than it probably should have. Evals are, by any reasonable measure, the most important quality discipline in AI product management. They are the thing standing between a model update and a customer discovering the regression for you. And yet running underneath the O'Reilly piece and the Reganti and Badam episode is something closer to fatigue: people who had built eval suites, run them for a year, and quietly noticed that nothing they learned from them had changed anything they did.
Twelve months ago the problem was that almost nobody had evals at all. That was the case I originally made in my AI Fluency series, and I still think it was right at the time. Enough teams took that kind of advice that the failure mode has moved. It is now uncommon to find a serious AI product with no eval suite. It does seem to be very common to find one with an eval suite nobody can interpret, that has never caught anything, and that no decision has ever hung on. The dashboard is green. The number went from 0.71 to 0.74. Nobody knows what either number means, and nobody would have done anything differently if it had gone the other way.
This is looking a lot like "eval theatre". And after spending a few months researching this, I think the backlash is really a reaction to it - not to evals themselves.
The backlash is not anti-eval
It's worth being specific about what the pushback actually says, because the noisy version of it is a strawman.
The people making the strongest case for evals are also the ones most aware of how badly they get done. Shreya Shankar puts it plainly:
People have been burned by evals in the past. People have done evals badly, so then they didn't trust it anymore, and then they're like, "Oh, I'm anti evals."
That's a reaction to a bad implementation, not to the idea.
The substantive critique is narrower and more interesting. Kiriti Badam frames it as a false dichotomy - the choice is not evals or production monitoring, and there is no reason to trust either extreme. His argument is that it gets genuinely hard to build an evaluation dataset covering all the kinds of interaction customers will actually have with your product. That is not a claim that evals are useless. It is a claim about the limits of what a pre-deployment test set can represent, and it's obviously true the moment you have watched real traffic hit something you tested carefully.
The research supports the modest version of that claim rather than the sweeping one. OpenAI's alignment team notes that evidence grounded in real usage is less vulnerable to models behaving differently simply because they are being tested - a caveat I skipped over the first time, and then had to go back to.
Because that caveat is where things got genuinely odd. Anthropic ran Claude Opus 4.6 through BrowseComp, a benchmark that tests whether a model can dig up information that is genuinely hard to find on the web. On two of the 1,266 questions, the model exhausted its legitimate searching and, instead of giving up, started reasoning about the question itself. It noticed the question felt contrived, worked out that it was probably sitting inside a benchmark, worked out which benchmark, found that benchmark's source code on GitHub, and decrypted its answer key - then went back to the open web and verified the answer properly anyway. The write-up is worth reading in full, but the part that matters here is what set it off. Not effort, since other runs burned far more compute without ever going near benchmark material, but the shape of the question. It recognised what a test looks like.
The second finding is stranger. Anthropic's interpretability team now have a tool that can read what a model is representing internally before it writes anything. Running Claude Sonnet 4.5 through a staged scenario designed to tempt it into blackmailing an executive, they could see it privately register the situation as "fake" and "fictional" - it had worked out that it was being tested at the same moment it worked out what it was being tested on. It then behaved impeccably. When they suppressed the internal patterns associated with recognising an evaluation and ran it again, it threatened blackmail some of the time. It had been passing the ethics test partly because it could tell it was a test.
I don't think either finding means very much for the eval suite of a normal B2B product. Nobody is decrypting your golden dataset. But together they made vivid something I had been treating as abstract: a test is a distinguishable situation, and the everyday version of that needs no cunning at all. Your test inputs are cleaner than real ones, and they were written by someone who already knew what the failure looked like. The gap between the artificial situation and the real one is not a rounding error you can design away with more care. It's a permanent property of testing.
So: evals are necessary, evals are insufficient, and a suite that measures the wrong thing is worse than none, because it makes you confident about something you haven't earned. The rest of this post is about telling the difference.
Four tests
These are the four questions I think we should be asking about any eval suite. They are ordered by how quickly they could embarrass you.
1. Can it detect a known regression?
This is the cheapest test and I don't think many teams run it. Take a failure you have already seen and fixed. Reintroduce it, deliberately, in a branch. Run the eval suite. Does the number move?
If it doesn't, you don't have a measurement. You have a number that varies for reasons unrelated to quality.
The most common way to fail this test is a heuristic evaluator: something that pattern-matches keywords, counts how many source documents were cited, or flags boilerplate phrases, and reports the result as a score. There is a case study I read, where an AI agent was pointed at a customer support chatbot to audit it. The first thing the agent did was read the team's evaluator.py, conclude that the keyword matching and reference counting carried no signal about whether users were being helped, and rewrite it before running a single evaluation. Replacing it with a real judge scoring relevance, accuracy and helpfulness per turn immediately surfaced problems the original approach could not have seen.
The numbers had looked like evaluation metrics the whole time. That's precisely the problem. Heuristic scores are indistinguishable from real ones until you test whether they respond to anything.
2. Has it been calibrated against human labels?
An LLM judge that has never been checked against a person is a hypothesis, not an instrument. Judges have well-documented biases - position bias, verbosity bias, a marked tendency to be generous towards model-generated text. Anthropic's own work on harness design for long-running agents is blunt about it: out of the box, Claude is a poor QA agent. In early runs it would identify a legitimate issue, talk itself into deciding it was not a big deal, and approve the work anyway. It took several rounds of reading the judge's logs and finding where its verdicts diverged from a human's before it graded sensibly.
The calibration is not complicated. Score a sample by hand. Score the same sample with the judge. Look at the disagreements rather than the correlation coefficient. If the judge is consistently more generous than you by fifteen points, that's a fixable offset. If it disagrees in different directions on different cases, your rubric is ambiguous and you are measuring noise.
Skip this and you optimise towards a metric that doesn't track quality - and you'll get exactly what you optimised for. High scores on paper, bad outputs in production.
3. Did a domain expert shape it?
This is the one PMs are best placed to fix, and the one I assume is most often skipped.
The instinctive move, when a failure mode looks subtle, is to reach for an LLM judge. The best example of this is Teresa Torres's eval work on her AI interview coach. The failure mode was the coach suggesting overly general interview questions - "tell me about your typical morning routine" - instead of behavioural ones. That sounds like judge territory: you have to understand what "general" means in the context of customer interviewing. Her domain expertise revealed something much simpler. A handful of red-flag words - usually, typically, in general - almost always mark the failure. A short code assertion caught nearly everything a human reviewer caught, with almost no false positives.
Hamel Husain's observation on that case is the part I keep coming back to: an engineer without her coaching expertise would have made both evals LLM judges, and would have shipped a slower, more expensive, less reliable suite. They would have done nothing wrong. They simply wouldn't have known the pattern was there to find.
The decision between a judge and an assertion isn't driven by how complex the task looks. It's driven by whether someone with domain knowledge can articulate a structural pattern the failure follows. When they can, write the assertion. When they can't, you need a judge - and see test two.
This is also the strongest argument for PM ownership of eval design I can think of. The usual argument is that PMs are accountable for quality, which is true but abstract. The concrete one is that eval quality is bounded by the domain expertise that went into it, and on most teams the person with that expertise is the PM.
4. Does a decision hang on the result?
The last test is behavioural, and it's the one that separates a measurement system from a ritual. Name the last decision that changed because of an eval run. A release held. A model swap rejected. A prompt reverted. A feature descoped.
If you can't name one, the suite is probably decoration. It's being run because running it is what responsible teams do, which is roughly the definition of theatre - the performance of a practice, detached from the function the practice exists to serve.
The tell is usually organisational rather than technical. Eval results that arrive after the ship decision has been made. A dashboard owned by nobody. A pass threshold that's never been enforced, because enforcing it would have blocked a release someone had already promised to a customer.
Why we produced theatre
It's worth asking why so many suites seem to fail these tests, because the answer is not that teams are lazy. Reading back over the last few months, three explanations kept recurring.
The first is that evals were sold as infrastructure, and infrastructure is something you procure. The tooling market is crowded, and the pitch is that you adopt a platform and get evaluation. But no platform can supply the two things that matter most - a golden dataset reflecting your actual task distribution, and a failure taxonomy built by someone with domain knowledge reading real outputs. Those are labour, not licences. What you can buy is the part that produces numbers, which is exactly how you end up with confident numbers about nothing.
The second is that error analysis is boring and looks unproductive. Reading two hundred real outputs and hand-labelling the ways they go wrong is unglamorous work that produces nothing anyone can demo. Shankar makes a related point about premature A/B testing: teams jump to experiments before doing any error analysis, then find, once they finally look at the data, that the errors are not the ones they assumed. The taxonomy has to come first, and building it is the tedious part everyone wants to skip. Husain's version - that the answer to "can't the AI just eval it?" is no - is a reaction to how badly people want that shortcut to exist.
The third is that we generalised from coding agents. Coding agents were the first AI products released at scale into the wild, and they have properties almost nothing else has: verifiable correctness, cheap ground truth, and builders who are also the users. Engineers dogfood coding agents relentlessly because they can't help it. Almost no other domain gets that. You do not have doctors dogfooding your clinical tool between patients. A great deal of eval advice is quietly coding-agent advice, and it transfers worse than it appears to.
Charity Majors' framing sits underneath all three. AI magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones; it won't solve for a lack of discipline, tooling gaps, or management disconnected from reality. Eval theatre is not an AI problem. It is the measurement theatre a team was already capable of, now with a dashboard.
Evals are one layer
The conclusion I have come to is not that evals matter less than I thought. It's that they were never going to be the whole system, and the discourse sold them as though they might be.
What actually holds quality up is a loop with four stages, of which pre-deployment evals are one. Evals catch what you already know how to look for. Production observability catches what happens at real volume, on real traffic, in shapes your test set doesn't contain. User feedback catches what neither notices, because it concerns whether people got what they came for. And the loop closes when what production surfaces goes back into the golden dataset as a regression case, so the same failure can't reach users twice.
Miss any one stage and the others silently take on work they can't do. Skip observability and your eval suite becomes the only thing standing between you and a distribution it was never designed to cover. Skip the feedback path and your dataset ossifies while user behaviour moves. Skip the loop back and every incident is a fresh incident, forever.
Assume, too, that any suite has a half-life. Evals are structurally reactive: they measure the system after it has changed, and they look for failure modes that were visible at design time. When a model crosses into a genuinely different capability regime, the suite doesn't fail loudly. It keeps returning numbers, and the numbers stop meaning what they meant. That argues for re-running the failure-discovery step on a cadence rather than treating the taxonomy as a fixed asset, and for budgeting new evals when a capability step-change lands rather than assuming the old ones transferred.
So the reframe I landed on is not "PMs should write evals". It is that PMs own the feedback loop - its design, its coverage, and whether anything actually happens when it fires. Writing evals is one task inside that. It's a smaller task than the discourse suggests, and it's by far the easiest one to fake.
The question I would take into your next quality review is not whether you have evals. Most teams do now. It is whether anything would be different if you didn't.
You're receiving this because you subscribed at stvpj.com. Reply to this email anytime.
