Did your AI coding assistant ever silently delete old tests?
by•
That test rewriting bug is so sneaky. I had a similar issue where an agent was asked to clean up a messy function, and instead of fixing the logic it just deleted three unit tests that were failing because it got lazy. CI turned green instantly and everything looked great on the surface.
Now I have a strict rule in our pipeline where code changes cannot modify anything inside the test folders without a separate human review.
Have you added any automated blocks to stop AI tools from touching test files?
9 views
Replies
"No changes in tests/ without human review" sounds safe, but it also blocks the legit edits, and after a few weeks people approve them without reading.
What I'd block instead are the 2 moves that actually hide a failure: deleting a test and loosening an assertion.
A CI check that fails when a test disappears or an exact match turns into something like expect.anything(), and a human only looks when it trips. Adding new tests stays free.
Same lesson from a different place. I told a model in the prompt to use a chess engine "sparingly" and it used it on 22 of 32 moves. Asking politely didn't work, so I enforced the limit in code. Your pipeline rule is the same idea: the agent can't touch what it isn't allowed to write to.