My AI quoted a score that never changed. The bug was in my “changed” list.
I built a rule into a text generator: it may only quote numbers that actually changed since last time. Code computes the changes and hands the model a list, so it has nothing to invent from. I was proud of that rule.
Then I read the code that builds the list. A category went onto it whenever the grading call returned any grade for it, even when the score hadn’t moved (100 to 100). The model was doing exactly what I told it: quoting a “change” that wasn’t one. The rule was fine and my definition of changed was wrong.
A likely second cause: when a user taps a short preset answer, the prompt also carries a memory of earlier check-ins. A bare answer plus a lot of context invites the model to fill the gap with habits the user never mentioned, and my specificity rules ask for specific details.
What I’m changing: compare the old and new score in code and only list categories that really moved, add prompt sentences that say a short answer means a short reply, and verify with a database comparison instead of reading the output. It’s written, not shipped yet.
The lesson I keep relearning: a guardrail is only as good as the data behind it. I tested the prompt for months and never tested the list.
How do you test the inputs you hand a model, not just the outputs? Do you assert on the exact payload, compare against a recomputed expectation, or something else?
Replies
Same lesson in a non AI part of my app: a safety switch that pauses music releases when the distributor rejects our token kept firing although the token was valid. The switch was fine. The input was not: 44 fake release records made for store screenshots looked like real submitted releases, so an hourly sync asked the distributor about an id it had never issued, got dozens of rejections in a row, and after that burst the distributor refused the good token for a while too. The fix was a flag on those fixtures. What I test now is the selection, not only the logic: which records go in, and could anything on that list be something the code never meant to send.
Same lesson here. My rule is that the model only writes the words, and a script builds every date, path and JSON field. But like you found, that only moves the risk into the script. What's helped me is checking the actual record in the database after the run instead of reading the output, the same thing you landed on. A reply that looks right proves very little.