RunAI Coder

Fifteen years ago a study went back through 1,694 merges in three open-source projects with runnable test suites. 76% merged clean and stayed fine. 16% hit a textual conflict, the kind with conflict markers. And another 1% and 6% merged clean, no markers, nothing to resolve, then broke the build or failed the tests. Of all the conflicts in that study, a third were ones the VCS had reported as a clean merge.

We wanted to see what it looks like with two coding agents in two git worktrees, which is what the coding CLIs' docs offer for running work in parallel. Ten times we made a base commit, gave two agents one worktree each and two tasks that never mention each other, one of which quietly depends on the other, one renaming a field the other one reads, one changing a file format the other one parses. Both agents ran the tests before finishing. All twenty worktrees were green.

Git stopped five of the ten merges with conflict markers. It passed the other five as clean, and the merged tree failed the combined suite in all five. The failing tests were always the second agent's own, unchanged, green minutes earlier in its own worktree. One pair we ran twice and got both outcomes: the model placed a new block of code one line away from the other agent's edit and git merged; placed it directly below, git flagged a conflict.

A merge compares hunks of each file with their common ancestor and knows nothing about callers or field names. A worktree guarantees each agent never reads the other's edits, which is exactly why the two changes first meet at the merge. Recent papers on agent PRs report 20-42% textual conflict rates, and even they admit that's just a lower bound.

In August one coding-agent CLI's release note took its task-tracking tools away from its newest models and kept them for the older ones, with an environment variable to reverse it. The note gives no reason. A component study posted in mid-September is the nearest thing to the arithmetic behind a call like that: a harness part is built to cover for something the model does badly, and the same part does a different job once the model stops doing that thing badly.

The study held an agent loop fixed and swapped parts across four models of different size. Its planning scaffold, a plan re-injected every turn, added 11.6 points on the repository benchmark for the smallest model, whose median run otherwise ended after five turns, two runs in three without an edit. For the largest it trimmed cost by about 30% with a two-point drop in success, because what it cut was verification the model would otherwise keep doing. A predefined tool set showed the same crossover: worth 15 points to the small model on the repository benchmark, while the large one did better there and 53% cheaper with bash alone, bundling several edits into one script.

Our reading: whether a part helps is a question about the model, the task and the budget; what the part costs, in tokens on every turn, is charged whatever the answer. A part that was right when it shipped can be wrong a model generation later without anyone touching it.

We tried the smallest version on our own setup: one small task, one model, fifteen runs, five under each of three tool lists. Every run passed the same four checks. With the task-tracking tools switched back on, the model was offered them in every run and called them zero. With bash alone it finished in a median of 10 turns instead of 17, and the first request carried a quarter of the context. We kept the typed file tools on anyway, because path permission rules attach to them and cannot see inside a shell script.

We asked a coding agent fifteen times to make one button blue on a page built to tempt it: Add to Cart and Checkout shared a class, a badge and the nav link shared the accent variable. Every run added an id-scoped rule. Checkout stayed green. Appending "Do not change anything else" changed one thing: the five runs with it wrote the hex inline where nine of the other ten had added two CSS variables. A path rule fencing off every file but the stylesheet never fired.

Real repositories look different. One benchmark pre-applied the fix to 200 issues and reopened them for five recent models in their vendors' harnesses. The right patch is empty; 35 to 65% of instances got an edit to executable code anyway, and in the authors' analysis of one model's failed traces, 87.1% had modified code unrelated to the issue. Task wording moved that figure a long way in both directions. Telling the agent to "edit the codebase" dragged one model's correct-abstention rate to 36.5%; reproduce first, then fix or abstain if nothing was wrong, lifted it to 88.5%. Their explanation: the agent acts on whatever it believes success means.

The harness side: switch on auto-accepted edits in both CLIs whose docs we read and the grant is the whole working directory. A path rule brings that down to files; one vendor's docs state that it applies to the built-in edit tools and recognized shell commands, and that a script opening a file itself is outside it. A selector inside a permitted file sits below every layer's resolution. Whether the blue landed on one button or on every primary button is decided by the sentence you typed and by whoever reads the diff.

When your agent overshoots, where does the boundary live in your setup today: the prompt, a rule file, a path rule, a sandbox, or the person reading the diff? Which of those has caught a change for you inside a file the agent was allowed to edit?