Two bugs I shipped past green checks that could not fail

Twice this year I shipped a defect straight past a check that was supposed to catch it. Both checks ran, both went green, and neither could have gone red.

The first was a "no horizontal overflow" check on a redesign. It passed over a real collision between a price and the note next to it, because the page had body { overflow-x: hidden }. That one line hides intra-grid collisions from the viewport, so the check was measuring a symptom the CSS had already suppressed.

The second was in a scanner I wrote to audit sites. It sampled the first N URLs from the sitemap, which sounds reasonable until you notice the head of a sitemap is the homepage and the top nav, the best maintained URLs any site has. It covered half a percent of a 107,717-URL sitemap and reported 1 qualified account out of 20. Random sampling found 5. My fixture sitemap had 9 URLs, so head sampling and random sampling were the same code path in the test.

That's the pattern. A self-test only encodes the failure modes I already thought of, which is exactly the set the check already handles, so it's near guaranteed to pass. It measures my imagination, not the check.

What I do now before trusting a check: inject the defect it claims to catch, watch it go red, then revert. If it stays green it's decoration. And look at the shape of the real input, not just the behaviour on a miniature of it.

Has a green check ever hidden something from you? What tipped you off?