2026-10-19
October 19, 2026
Before/after charts are the default evidence for AI tooling ROI, and they prove almost nothing. Compare a group that got the tool with one that didn't, or don't present the number.
Somewhere in your org this quarter, someone presented a slide that says cycle time dropped 25% after the AI coding rollout. The chart has a clean step down in the month the licenses went out. Everyone nodded, because the chart confirmed what everyone already believed.
Now ask what else changed that quarter. A reorg landed, two platform migrations finished, the hiring freeze lifted, a noisy legacy service was retired. Any of those can move cycle time. A before/after comparison attributes all of it to the one change that has a launch date and a champion. Greg Wilson, in his May 2026 post "Twelve Ways to Be Wrong About AI-Assisted Coding," lists this failure third: before/after comparisons without a control group. He lists eleven more, and most of them are on somebody's dashboard right now.
This is a program-management problem more than a data-science problem. The TPM is usually the person who assembles the rollout narrative, which makes the TPM the person best placed to insist that the narrative be falsifiable. The fix is cheap if you design it before the rollout and nearly impossible if you try to reconstruct it afterward.
Start with what the measurement community already knows about why these numbers mislead. Wilson's twelve are worth reading in full, but they cluster into four failures that a program lead can actually act on.
Failure one: no counterfactual. Before/after comparisons, volunteers versus non-volunteers, and "AI versus nothing" all share this flaw. Early adopters were often your most active engineers before the tool arrived (Wilson cites a Copilot study where users were more active even before the tool was introduced). And the realistic alternative to an assistant is not no help at all; it is documentation, a colleague, or thinking harder. Compare against that.
Failure two: measuring the easy half. Generation speed is trivial to instrument. Review time, debugging, security findings and accumulated complexity are not, so they quietly drop off the page. Wilson points to a study of Cursor adoption that found a transient velocity boost alongside a persistent increase in complexity. A short measurement window captures the boost and misses the bill.
Failure three: measuring the individual instead of the system. Faster code writing moves the bottleneck to review. LeadDev's June 2026 piece, "The 8 software engineering metrics AI broke" by Michael Hill, makes the same point about lead time: it can fall precisely because humans are verifying less. The metric improves while the thing it was supposed to proxy gets worse.
Failure four: metrics that now measure the tool. James Stanier, quoted in the same LeadDev piece, puts it bluntly: output is now nearly free, so volume measures the tool and not the engineer. Jellyfish's Nicholas Arcolano adds that lines of code has become an input metric that correlates with token spend. Anything that rises as a side effect of normal tool use cannot also be evidence the tool worked.
So what holds up? The LeadDev piece names three: time to recover, customer and business outcome metrics, and escaped defect rate. Escaped defects are the cleanest example. The share of bugs that reach production after passing verification cannot be inflated by generating more code without consequence, and it tells you whether verification is keeping pace with generation.
There is a second design principle worth stealing from Microsoft. The EngThrive framework, led by Brian Houck's research group and covered by Chris Stokel-Walker for LeadDev in May 2026, does not try to make metrics ungameable. It tries for what its authors call gaming alignment: if people optimize the metric, the result should still be useful. Researcher David Manheim's caveat is the one to keep: no single metric or small group of metrics is immune, so use the framework as a dashboard, and keep it away from pay and promotion, where pressure breaks fragile measures. Note also that EngThrive is aimed at teams and organizations, not individuals.
Put those together and you get a measurement brief a TPM can write on day one of any tooling program. Pick the outcome metrics before launch (escaped defects, recovery time, one customer-facing outcome). Pair each speed metric with a quality counterpart. Define the comparison group now, and stagger the rollout so one exists. Decide in advance how long you will measure, so the novelty period does not set the verdict. And write down the date you will retire each metric, because a metric that has served its adoption purpose becomes noise, a point Ankit Jain of Aviator makes in the LeadDev piece.
The uncomfortable part is political. A staggered rollout means some teams wait, and the teams that wait will ask why. Your answer is that the org is buying evidence, and evidence is cheaper than a second year of licenses nobody can justify or cancel.
Try this week. Pick one AI tooling claim currently in a deck, such as "cycle time down X%" or "PR throughput up Y%". Write two sentences: what else changed in that window, and what comparison group would isolate the tool. If you cannot name a group, add the caveat to the slide and propose a staggered rollout for the next team onboarding.
What it is. A comparison technique from empirical economics: measure the change in a treated group and the change in a comparable untreated group over the same period, then subtract. The classic application is David Card and Alan Krueger's 1994 study of fast-food employment after New Jersey raised its minimum wage, using neighboring Pennsylvania as the comparison.
When to use it. Any rollout where some teams get a tool, process or policy before others: AI assistants, a new review process, a platform migration, a meeting-cadence change.
How to run it:
When NOT to use it. When the two groups were already diverging before the change, or when teams are so interdependent that the untreated group is affected by the treated one (shared review queues, for example).
Example: two otherwise similar teams both see cycle time fall 15%; only one got the assistant. The tool's measured contribution is roughly zero, and the rest was the quarter.
The 8 software engineering metrics AI broke — Michael Hill (LeadDev, June 2026) walks through why deployment frequency, lead time and activity metrics now partly measure tool usage, and which three survive; the best single reference to hand a skeptical VP.
Is Microsoft's EngThrive framework immune to Goodhart's law? — Chris Stokel-Walker (LeadDev, May 2026) on "gaming alignment" and the Bad Developer Days metric, plus a useful outside critique of any single-framework approach.
Twelve Ways to Be Wrong About AI-Assisted Coding — Greg Wilson's checklist of measurement mistakes; run your current AI dashboard against it before your next review.
"When a measure becomes a target, it ceases to be a good measure."
— Marilyn Strathern, paraphrasing Goodhart's law (1997)
Don't miss what's next. Subscribe to Critical Path: