2026-10-26
October 26, 2026
Self-reported AI time savings nearly doubled in a year. The innovation ratio moved one point. Your metric is missing a column.
DX's Q2 2026 data has a number that every engineering review deck will quote: median self-reported AI time savings rose from 3.3 hours per week (Q3 2025) to 6.1 hours per week (Q2 2026). Multiply by headcount, multiply by a loaded rate, and you have a business case that writes itself.
Now read the other numbers from the same dataset. The innovation ratio, the share of time spent on new capability rather than maintenance and overhead, went from 57% to 58%. DX's Developer Experience Index fell from 67 to 65, the first downward move in the index's history. Change confidence dropped 6.1%. Median pull request size went from 42 to 72 lines. Median quarterly AI spend per organization went from roughly $1.5K to nearly $44K. If hours really were saved, they went somewhere, and it was not into new work.
That is the whole argument. A gross savings number is a claim about a mechanism, not evidence of an outcome. As a senior TPM or tech leader, your job in the next budget review is to be the person who asks for the second column: where did the hours land? The rest of this issue is how to build it.
Start with what the number is. Self-reported hours saved is a perception metric. It is useful, and it is also the weakest class of evidence you have. The METR study of experienced open-source developers in 2025 found them 19% slower with AI tools while believing they were about 20% faster. That does not make the DX survey wrong; the populations differ and DX's user base is self-selected toward companies already investing in developer experience. It does mean "developers say they save six hours" and "the organization gets six hours back" are separate claims, and only the first has been measured.
Then find the leak. Time saved at the keyboard has to survive three transfers before it becomes output: it has to survive review, survive integration, and then be redeployed to work that matters. Each transfer has a known failure mode in the data. Review: Faros AI's 2025 telemetry paired a 98% jump in merged PRs with a 91% rise in PR review time. Integration: DX reports PRs nearly doubling in size while change confidence falls, and the 2025 DORA report found AI adoption now correlates with throughput but still negatively with delivery stability. Redeployment: a flat innovation ratio says the freed capacity is being absorbed by something other than new capability, whether that is rework, coordination, or simply more output of the same kind.
Price the friction in the same currency. DX estimates each point of its index is worth about 10 hours per engineer per year, so a two-point drop implies roughly 20 hours of new friction per engineer annually. DX itself calls that a modeled estimate, and on paper it is small next to roughly 6 hours a week of claimed savings. Do not read that as reassurance. Read it as a prompt: the gross figure and the friction figure are measured by different instruments, one by survey self-report and one by a model, and neither is a direct measure of delivered work. A ledger with a large soft number on one side and a small modeled number on the other tells you nothing yet.
Use the three-column ledger. For any AI rollout claim, require: (1) the gross claim, with its instrument named; (2) the downstream cost, meaning review time, rework within 30 days, and change failure rate, each measured from system data; (3) the destination, meaning what the freed capacity produced, tracked through something like innovation ratio, roadmap throughput, or cycle time on a fixed class of work. If a team can only fill column one, they have a hypothesis. That is fine. Label it a hypothesis.
Watch the spend line as an input, not a result. Quarterly AI spend moving about 28x in the tech sector is a cost you chose. It belongs on the ledger as an input, and the ratio you want is outcome per dollar on a stable unit of work, not tokens consumed or percent of code AI-generated (DX reports that share rose from 34% to 52% in a single quarter, which tells you adoption, not value).
The leadership move. Atlassian's research across more than 12,000 knowledge workers found only 6% of the 170 Fortune 1000 executives surveyed could point to organization-wide AI ROI. That is not an indictment of the tools. It is a description of the measurement gap you can close in your own program, with a ledger and a quarter's discipline.
Try this week. Pick the one AI productivity claim currently in your leadership deck. Write three lines under it: the instrument behind the number, the review time and 30-day rework rate for the same teams over the same period, and the share of freed time that went to roadmap work. Any line you cannot fill is the measurement project for next quarter.
What it is. Amazon's practice of separating output metrics (orders, revenue, which you cannot directly manipulate) from controllable input metrics (the things a team can directly change, such as selection, price, and availability), then checking every week that the inputs actually move the outputs. Outside Amazon, this maps to leading versus lagging indicators.
When to use it. When you are about to pick the metric a team will be held to, especially for an AI rollout, a platform migration, or any program where the obvious number is an activity count.
How to run it:
When NOT to use it. When the output is already directly observable and fast, or when you lack the data to ever test whether the input predicts the output.
Applied to AI: "percent of code AI-generated" is the detail-page count. "Share of AI-assisted changes merged without rework in 30 days" is closer to the in-stock-and-shippable version.
The State of AI Impact in Engineering: Q2 2026 — DX's data on time savings, the first-ever DXI decline, and a flat innovation ratio; the best current dataset for the "where did the hours go" question, with the caveat of a self-selected sample.
Atlassian research on the AI efficiency paradox, via diginomica — Pairs DX's engineering data with Atlassian's finding that only 6% of surveyed executives can point to organization-wide AI ROI; useful framing for any budget conversation.
The Engineering Leadership Report 2026 — LeadDev's survey of 600 engineering leaders: 37% doing more hands-on technical work, one in three managers considering a move back to IC, 45% working more hours than a year ago. Context for who is reading your metrics.
"When a measure becomes a target, it ceases to be a good measure."
— Marilyn Strathern, 1997, restating Goodhart's law
Don't miss what's next. Subscribe to Critical Path: