Researchers at Purdue, Microsoft and Chicago built six ways of handing a coding agent the same abilities, then set it loose on real GitHub issues, 11,700 runs across three models. One agent got a plain shell and nothing else. One got a small set of named operations: view a file, replace a string, create a file. One got a search box that takes questions in English. One wrote Python instead of calling tools. Two got somewhere to jot down their thinking. None of it let the agent do anything it couldn't already do from a shell prompt.

The share of issues fixed came out much the same across all six. Then everything a team actually cares about moved.

"Even when two tool setups support similar tasks, they can still lead the model to behave differently because they present similar underlying capabilities through different interfaces."

Letting the agent write Python instead of firing tool calls one at a time got the same results with about 40% fewer steps and roughly half the token bill. The saving came from fewer round trips, not cheaper ones. Every step re-sends the whole conversation so far, so the way to make an agent cheap is to make it stop asking.

The small set of named operations was the only setup that made repeated attempts at the same task more consistent for every model, mostly by cutting the sloppy stuff: broken edits, malformed commands. The weakest model gained most. The same tidy interface cost the two stronger ones about a fifth more in tokens, because a capable model chains several operations into one shell command and neat little tools chop that back into separate turns.

"This pattern suggests that the efficiency effect of tool architecture depends not only on what operations the interface makes available, but also on whether its granularity matches the actor's natural interaction style."

The two note-taking tools barely registered. They were deliberately bare, somewhere to write and nothing more, and the models used them to restate what they were going to say anyway.

"In other words, the scratchpad frequently acts as an explicit channel for restating existing reasoning, rather than as a tool that elicits substantially different intermediate thoughts or planning structure."

That leaves a measurement problem. All six setups scored the same on the metric most teams judge them by: did it fix the issue. A team A/B-testing its harness on pass rate will conclude the harness doesn't matter, while the two things engineers actually complain about are the two that moved. Nobody experiences an average success rate. They experience whether it worked when they asked, and what it cost. An agent that succeeds intermittently gets abandoned long before anyone files a ticket.

Then there's the finding a team should argue about: the best interface depends on which model is behind it. Cheap models want guard rails; strong ones want to be left alone with the shell. A harness tuned against a flagship doesn't transfer downward, so a team switching to a cheaper tier to control spend may land on exactly the wrong tooling for it. "Our tool design improves agent performance" needs a follow-up: on which model, and does that hold on ours?

Two ways this will get over-read. It isn't evidence that semantic code search is snake oil: the search box here had no index behind it, just grep with a friendlier front door, and it still got the agent reading more of the right code, so some of the credit given to expensive retrieval infrastructure belongs to the interface. And the broader reading barely changed the patches the agent produced. Exploring more is not a result.

Sources

1. Xu, X., Saghir, H., Wu, Q. et al., "The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior" — https://arxiv.org/abs/2608.11386