Connect with the Product Hunt community

Start a thread, join a discussion, stay in the loop.

Start new thread
trending
General

p/general

by

•
The conflict git cannot see is the one parallel agents produce

Fifteen years ago a study went back through 1,694 merges in three open-source projects with runnable test suites. 76% merged clean and stayed fine. 16% hit a textual conflict, the kind with conflict markers. And another 1% and 6% merged clean, no markers, nothing to resolve, then broke the build or failed the tests. Of all the conflicts in that study, a third were ones the VCS had reported as a clean merge.

We wanted to see what it looks like with two coding agents in two git worktrees, which is what the coding CLIs' docs offer for running work in parallel. Ten times we made a base commit, gave two agents one worktree each and two tasks that never mention each other, one of which quietly depends on the other, one renaming a field the other one reads, one changing a file format the other one parses. Both agents ran the tests before finishing. All twenty worktrees were green.

Git stopped five of the ten merges with conflict markers. It passed the other five as clean, and the merged tree failed the combined suite in all five. The failing tests were always the second agent's own, unchanged, green minutes earlier in its own worktree. One pair we ran twice and got both outcomes: the model placed a new block of code one line away from the other agent's edit and git merged; placed it directly below, git flagged a conflict.

A merge compares hunks of each file with their common ancestor and knows nothing about callers or field names. A worktree guarantees each agent never reads the other's edits, which is exactly why the two changes first meet at the merge. Recent papers on agent PRs report 20-42% textual conflict rates, and even they admit that's just a lower bound.

0

General

p/general

by

•
A harness feature that expires when the model improves

In August one coding-agent CLI's release note took its task-tracking tools away from its newest models and kept them for the older ones, with an environment variable to reverse it. The note gives no reason. A component study posted in mid-September is the nearest thing to the arithmetic behind a call like that: a harness part is built to cover for something the model does badly, and the same part does a different job once the model stops doing that thing badly.

The study held an agent loop fixed and swapped parts across four models of different size. Its planning scaffold, a plan re-injected every turn, added 11.6 points on the repository benchmark for the smallest model, whose median run otherwise ended after five turns, two runs in three without an edit. For the largest it trimmed cost by about 30% with a two-point drop in success, because what it cut was verification the model would otherwise keep doing. A predefined tool set showed the same crossover: worth 15 points to the small model on the repository benchmark, while the large one did better there and 53% cheaper with bash alone, bundling several edits into one script.

Our reading: whether a part helps is a question about the model, the task and the budget; what the part costs, in tokens on every turn, is charged whatever the answer. A part that was right when it shipped can be wrong a model generation later without anyone touching it.

We tried the smallest version on our own setup: one small task, one model, fifteen runs, five under each of three tool lists. Every run passed the same four checks. With the task-tracking tools switched back on, the model was offered them in every run and called them zero. With bash alone it finished in a median of 10 turns instead of 17, and the first request carried a quarter of the context. We kept the typed file tools on anyway, because path permission rules attach to them and cannot see inside a shell script.

0

First
Previous
•••
118119120
•••
Next