Top Highlights
- A passing test isn’t the whole story: agents can reach the same result after unnecessary edits, reruns, and extra turns—work a pass/fail score misses.
- The benchmark simulated agent decisions across about 50,000 runs on real repositories; it did not test Claude Code, Codex, or other real agents.
- The simplest safeguard mattered most: blocking “finish” while tests fail improved success; adaptive rules reduced some wasted work, but often added more turns than they saved.
- Measure both the journey and the patch: fewer leftover changes can still mean more total churn. Treat these findings as simulation results, not predictions about real agents.
A Passing Test Can Hide Extra Work
A coding agent can solve a task and still take an inefficient path. In one example, tests passed on turn four. The agent then made six more turns, changing files that did not need changes. Its final patch altered eight lines across three files; the reference fix changed only two lines in one file.
A basic pass-or-fail score treats that run like a clean, five-turn solution. Yet the extra edits create more review work and can leave unrelated changes in the patch. So a passing test tells us whether the code works—not how much effort the agent wasted getting there.
More Guardrails Can Help — At a Cost
In a simulation of 50,000 runs, a basic harness let agents stop even when tests failed. A completion check prevented that behavior and raised success rates. Other rules asked the agent to review changes or avoid repeated actions.
However, those safeguards added turns. In some simulated cases, the harness reduced wasted work for less reliable agents, but the full run still took longer. One rule also removed unnecessary lines from the final patch while increasing the measured amount of work along the way. The metrics captured different costs.
Measure the Journey, Not Just the Result
The study did not run Claude Code, Codex or another real coding agent. It simulated agent decisions while running edits and tests in real repositories. Its results, therefore, show what can happen under those assumptions—not how real agents will perform.
Still, the lesson applies to adoption: teams should track test results, turns, repeated commands and changes that do not survive into the final patch. Then they can compare efficiency without rewarding an agent simply for stopping early. A green test matters. So does the work behind it.
Discover More Technology Insights
Learn how the Internet of Things (IoT) is transforming everyday life.
Discover archived knowledge and digital history on the Internet Archive.
AITechV1
