The model writing your tests is not a second opinion, it's the same opinion twice
Most vibe coding advice says to have the agent write the tests alongside the code. I think that's the most expensive habit in the whole loop.
The code and the test come out of the same reading of what you asked for. If that reading is wrong, both are wrong in the same direction and the suite goes green, which is worse than having no suite at all, because now you've got a receipt.
We hit a version of this on Arteza with a balance that went stale. The check was green the entire time, and not because the check was working. Nothing had ever handed it a wrong balance to catch, so green meant untested, not passing.
What I do now is write the assertion myself first. Badly if necessary, three lines is fine. The model implements against it and tidies it up after. The statement of what correct means has to come from outside the thing being graded, and the agent can't be both.
If you vibe code anything that touches money or state, what's your actual habit? Hand written assertions first, or generated tests you promise yourself you'll read properly?
Replies
A coding agent fixed a broken parser test by changing the assertion from checking the exact payload to just checking that an object was returned. The suite stayed green for days before anyone noticed the parser was dropping nested fields.
@konstantin_tikhaev That's the one that scares me most, because the agent edited the grader to pass its own exam and nothing in that diff looks alarming. Checking that an object came back is the test equivalent of a 200, it only proves the wire worked. We keep assertion files outside what the agent can touch now, it can add tests but not loosen one that already exists.
@asadmalik901 Existing tests have to be immutable. The moment an agent can edit both the implementation and the validator, green just means the model agreed with itself.
Agree, and there's a second version of this: tests that can't see the thing at all. My API's pytest suite runs on SQLite, which has no Postgres triggers or row-level security. Green every time, and it never touched the parts most likely to leak data. Those I check by hand with psql against a real Postgres, as anon and as a logged-in user. A green suite only tells you about what it could run.
@wwenioaguiar The SQLite one is worse than a wrong test because the suite isn't wrong, it's blind, and blind reads exactly like passing. Row level security especially, since the failure is a row showing up for the wrong user and there's no exception anywhere for a test to catch. I'd take ten slow tests against a real Postgres container over four hundred fast ones that can't see the policy.
@asadmalik901 Agreed, and a real Postgres container still has a blind spot: it tests your schema, not your production database. I had a case where every migration was correct and local Supabase was clean, but the hosted project had broad default grants from provisioning that no migration ever revoked. A column meant to be private was readable by anonymous users. Nothing failed anywhere, locally or in prod. The fix was revoking everything and re-granting only what the schema files declare. Before applying it, I replayed every real frontend query against production as anon and as a logged-in user inside BEGIN...ROLLBACK. Now I compare the live grants (information_schema.role_column_grants) against the schema files after setting up any hosted project.
hand written assertions, and only for the handful of paths where being wrong costs someone money or trust. we build voice agents at Dial, and the rule that's held up is: anything where the agent's output becomes someone else's next action without a human in between gets a human-written check first. everything else, I let the generated suite ride because a false negative there just means a slightly worse reply next turn, not a wrong invoice or a dropped call. the expensive mistake is applying the same rigor everywhere, that's how people burn a week hand writing assertions for a button color
@galdayan Fair, and the button colour line is right. Where I'd push is that the money path is wider than it looks. Our bad balance events weren't in payment code, they came out of a spend function three steps away that wrote the numbers the balance got summed from, and nothing about that file said money. So the rule I landed on is anything that writes a number someone else reads later, not anything near checkout.
@asadmalik901 that's a cleaner rule than mine, it catches the spend function case where "near checkout" wouldn't. the thing I'd stress test it against is who reads the number. our call-duration log that a billing job sums is obviously covered by your rule, a person eventually looks at that total. but we also write a confidence score that routing logic reads to decide whether to retry a call, no person ever sees that number directly, it just steers behavior. does "someone reads later" include another process reading it, or is the rule really about anything a human eventually looks at, even indirectly?
Mid-task the agent lost context, recreated an existing helper under a new name two files over, and wrote unit tests against the duplicate. The suite was completely green, but production code never touched either of them.