Your bug reports measure how loud your product is, not how correct it is
Three things went wrong on Arteza this year and not one was reported by a user.
A spend function wrote bad balance events, so the ledger drifted and the cap I thought I had wasn't the cap. Nothing threw, nothing alerted. Our sign-in gate promised a credit amount that didn't match what the account actually got, and I only found it by opening every modal in the product one at a time and reading what it promised. And before launch I said the failure I expected from 111 one click apps wasn't a crash, it was an app returning something plausible and slightly wrong that nobody would bother to report.
Same pattern in all three. A crash gets reported because it interrupts someone. A wrong number interrupts nobody, so it sits there and the inbox stays clean, and a clean inbox reads like a working product.
So the bugs you hear about are selected for how annoying they are, not how expensive they are. The expensive ones are quiet by construction.
I don't have a method beyond reading every screen by hand, which stops working the moment the product is bigger than one head. If you've found something that surfaces the quiet class, I want it.
Replies
The quiet class I keep running into is text an AI agent reads. I looked at 86 version updates of popular MCP servers and 40 changed what a tool says it does or accepts, mostly rewritten parameter descriptions. The agent reads the new text and behaves a bit differently. Nothing crashes, nobody files anything. Gal's second-number idea is the only thing I've found that works for it: keep what you approved and diff it against what's there now.
@neelagiri_chettiyar 40 out of 86 is worse than I'd have guessed, and it reframes the thing for me. A tool description is prompt text you didn't write, didn't review, and that ships on someone else's release schedule. Pinning a version number doesn't save you when the meaning inside that version moved. The diff you're describing only works if it blocks the upgrade rather than notifying someone, because a notification about changed wording is exactly the kind of thing people skim and approve.
@asadmalik901Â That's where it ended up. It doesn't block the upgrade it blocks the call. The new version can install but the agent can't use a tool whose description changed until someone has read the new text. The part that still only notifies is a change in the schema with no wording moved. I haven't decided whether that should block too. The worry is stopping a working flow over a type change nobody will act on.
this is the same spend function bug from your test-writing thread, right? which is kind of the point, "reading every screen by hand" and "writing the assertion yourself first" are the same fix wearing two names, you're forcing a number to have an external check instead of trusting whatever produced it. the thing that's worked for us on the quiet class specifically is reconciliation jobs that run on real data after launch, not test data before it. pick two numbers that should independently agree (balance vs sum of ledger events, promised credit vs granted credit) and alert on drift between them, continuously, not just at write time. it doesn't catch everything but it catches exactly the "plausible and slightly wrong" shape you're describing, because that shape always leaves two things that should match and don't. do your three bugs have an obvious second number they should have agreed with, or were some of them genuinely invisible even in hindsight?
@galdayan Two of the three, yes, and embarrassingly so. Balance against the sum of ledger events would have caught the spend function the day it shipped, and promised credit against granted credit is a one line check I never wrote because it felt too obvious to be wrong. The third has no second number, which is why it's the one that still bothers me. An app returning a plausible image or a slightly off voice has nothing numeric to disagree with itself, so reconciliation has no purchase on it at all. Best I've got there is making a wrong result cheap to report, and that's a much weaker guarantee than drift alerting.
Your third one is the kind I keep running into. In my AI chess/Go arena, the wrapper around a Go engine parsed the engine's error replies as moves. GTP marks success with "=" and failure with "?", and the code never checked which one came back, so "? illegal move" went downstream as if it were the engine's answer. Nothing crashed, nothing logged. No user would ever report that. It looks like a weak move, not a bug. I only found it reading the protocol handling line by line, and now there's a test that feeds it a "?" reply and expects an error.