Skills are the small markdown files teams now write to make coding agents better at recurring jobs: here's how we deploy, here's how our test harness works. The working assumption is that a skill teaches the agent something it didn't know. A team from Princeton, Stanford, UC San Diego and elsewhere ran thousands of matched runs, holding the task and the pool of past traces fixed and varying only whether that experience was distilled into a skill, then hand-labelled what changed.

Mostly, the agent didn't learn anything. Skills worked by stabilising how the agent acted rather than by supplying facts it lacked: pinning down a procedure accounted for around two-thirds of the cases where a skill helped, and injecting missing knowledge for about one in twenty. The failure types that skills nearly eliminated were all operational: botched environment setup, wrong output format, servers left running. The ones they barely touched were the thinking failures, like a wrong algorithm, or checking the code by reading it instead of running it.

That points at a narrower job than most skill libraries are written for. On this evidence a skill earns its place when it encodes the thing your agent gets wrong the same way every time, which is a runbook, not a tutorial. Most libraries that have been growing a while will have entries that are really just documentation, and they aren't paying for themselves.

The abstraction also brought its own failure mode. Agents applying a skill mechanically, missing a condition or carrying over an assumption that no longer held, showed up in roughly one run in ten, something that essentially never happened without skills.

"A skill is not self-executing: the agent must decide whether it applies, which parts to follow, how to adapt it, and when to abandon it."

Growing the library made this worse in a way that's easy to miss. As the candidate pool went from 5 skills to 100, the share of the skills an agent opened that were the ones labelled correct for the task fell from about 30% to about 3%, while its task success rate stayed flat. The researchers read this as near-miss skills still supplying useful procedural support. That's plausible, but their design records which files were opened and whether the task passed; it can't show a wrong-but-related skill is what carried the run. Flat success fits just as well with the library mattering less either way.

"Thus, exact ground-truth skill invocation is neither sufficient nor strictly necessary for success."

The more useful read is a warning about measurement. Retrieval accuracy is the easy thing to instrument in a skill system, and on this evidence it doesn't predict whether the work gets done, so a vendor demo that leads with how precisely the agent selects skills is showing off the metric that decoupled from outcomes first. The researchers also found that skills distilled from failed runs only helped when whoever wrote the skill knew those runs had failed. A self-improvement loop that hoovers up transcripts without recording which ones ended badly is accumulating a hazard, not a library.

The study covers terminal-style, tool-using work, so none of this speaks to long-horizon or open-ended tasks. But the direction is clear enough: growing the library is the cheap move, and it was the one that bought nothing.

Sources

1. Jiang, Z., Huang, F., Xing, H. et al., "Demystifying Agent Skills: Why They Work—Until They Don't" — https://arxiv.org/abs/2608.14036