Craig Bennett

Three things went wrong on Arteza this year and not one was reported by a user.

A spend function wrote bad balance events, so the ledger drifted and the cap I thought I had wasn't the cap. Nothing threw, nothing alerted. Our sign-in gate promised a credit amount that didn't match what the account actually got, and I only found it by opening every modal in the product one at a time and reading what it promised. And before launch I said the failure I expected from 111 one click apps wasn't a crash, it was an app returning something plausible and slightly wrong that nobody would bother to report.

In the early days, our product had a single, crystal-clear value proposition. Users logged in, knew exactly what to click, and achieved their goal in seconds. Because the core experience was focused, conversion rates were high and onboarding was practically effortless.

Then came the desire to expand. We started building every feature users requested, adding customizable settings, and creating multiple workflow options. Instead of delighting people, the experience deteriorated. More options split user attention, clutter ruined the clean interface, and new features introduced unexpected edge-case bugs. We added value on paper, but in reality, we just added friction.

How do you strategically say "no" to feature requests so your product stays lean, intuitive, and effective as it scales?

We asked a coding agent fifteen times to make one button blue on a page built to tempt it: Add to Cart and Checkout shared a class, a badge and the nav link shared the accent variable. Every run added an id-scoped rule. Checkout stayed green. Appending "Do not change anything else" changed one thing: the five runs with it wrote the hex inline where nine of the other ten had added two CSS variables. A path rule fencing off every file but the stylesheet never fired.

Real repositories look different. One benchmark pre-applied the fix to 200 issues and reopened them for five recent models in their vendors' harnesses. The right patch is empty; 35 to 65% of instances got an edit to executable code anyway, and in the authors' analysis of one model's failed traces, 87.1% had modified code unrelated to the issue. Task wording moved that figure a long way in both directions. Telling the agent to "edit the codebase" dragged one model's correct-abstention rate to 36.5%; reproduce first, then fix or abstain if nothing was wrong, lifted it to 88.5%. Their explanation: the agent acts on whatever it believes success means.

The harness side: switch on auto-accepted edits in both CLIs whose docs we read and the grant is the whole working directory. A path rule brings that down to files; one vendor's docs state that it applies to the built-in edit tools and recognized shell commands, and that a script opening a file itself is outside it. A selector inside a permitted file sits below every layer's resolution. Whether the blue landed on one button or on every primary button is decided by the sentence you typed and by whoever reads the diff.

When your agent overshoots, where does the boundary live in your setup today: the prompt, a rule file, a path rule, a sandbox, or the person reading the diff? Which of those has caught a change for you inside a file the agent was allowed to edit?