What's the smallest, dumbest thing that made you completely lose trust in an AI agent mid task?
by•
It doesn't even have to be a big dramatic failures, more the small moments where something clicked and you went from trusting the output by default to double checking everything. For me it was watching an agent confidently rename a function across twelve files, then leave the original function untouched in a thirteenth file it apparently didn't search, with zero indication anything had been missed. It wasn't even a hard case, the file just wasn't in the directory it happened to grep first.
What was your moment? And did it actually change your workflow afterward, or did the trust creep back in after a week like it always seems to for me?
521 views
Replies
An agent wrote a "benchmark" report with precise numbers. It was totally believable. Plus we spent a full day chasing pipeline anomalies before we realized. It had hardcoded placeholder values in the collection scritp instead of actually querying the system.
It wrote a fix and a new test to prove it worked. I reverted the fix out of habit to watch the test fail. But it stayed green). And the agent wrote an assertion that did not test the faliure path at all, just manufactured a passing suite
mine was almost the same shape as yours, one layer down. it fetched a page, parsed it cleanly, and told me a thing wasnt there. the fetch returned 200 and the parse was fine. the page just renders its content client side, so the server response only ever carried a subset of it, and everything downstream of that was confidently wrong. what got me was that nothing in the chain failed. a crash i can handle. this one returned success at every single step and produced an answer i repeated to someone else before i checked it. it did change the workflow. i stopped treating a successful fetch as evidence of anything and started confirming against the rendered page instead. your thirteenth file is the same bug wearing different clothes, the tool reported on what it searched rather than on what existed, and nothing in the output tells you which one you got.
Neeraj's follow-up is the sharp part: same confidence whether it checked one file or thirteen. The oldest measurement of that gap I know is from 1985. Lawyers on a document search stipulated they needed at least seventy five per cent of the relevant documents and worked each request until they believed they had it. Measured mean recall was about twenty per cent.
So I stopped reviewing my own search terms and built the check to write its test set in steps that cannot see the term list.
Two of them, and the fix in both cases was to stop relying on the agent remembering.
The small one: Claude Code ran `npx asar extract-file app.asar index.js` from the wrong directory, so the extracted file landed on my own `index.js`, the entry point of an Electron app. No error, no warning. The app simply stopped starting. What actually ended the trust was the same agent doing it again a week later, after I'd explained it, because it had no memory of the first time. The fix went into the repo rather than the instructions: the build now refuses to run if `index.js` doesn't look like itself, and destructive shell patterns are rejected by a pre-tool hook before the command is allowed to run at all.
The expensive one was Prisma, which is what Claude Code reaches for first if you let it pick. `prisma db push` decided the schema had drifted, reset the database and took the data with it. Nothing malicious, that is what the command is for. After that, `db push` and `migrate` are banned outright in that project's config, a `pg_dump` is mandatory before anything touches the database, and we moved most projects onto Drizzle with hand-written numbered SQL migrations that a deploy script applies. Nothing in the chain now generates destructive DDL on its own, which means the agent is no longer the thing standing between me and an empty table.
To your second question, yes, the trust comes back within a week, exactly as you describe. I stopped treating that as a discipline problem. Each specific failure gets turned into something that can't happen again, and the trust can come back as much as it likes after that.
Confidently telling me it had done something it hadn't. Not a wrong answer — a done that wasn't done. One of those and I start checking everything, which costs more than doing it myself.
Being wrong is fine. Being wrong in the same tone as being right is the problem.
Mine was not the agent. It was me. A review of my own write-up caught that the tier number routing the report was a product of means, while the per-risk number sitting next to it in the same graph was a mean of products. Two estimators, one graph, and the coarser one decided the branch.
The output looked fine. High tier at CHF 60,750 against Medium at CHF 538,667, and nothing on the page said the two figures were computed differently.
Trust crept back for me too.