I ran a contrast audit on our own homepage today and got 51 failures. The number was nonsense.
51 failures on one page is a redesign. I was about to write it up as one.
Then I looked at the instrument instead of the site. Our CSS uses lab() colour values. My script pulled the numbers out of the string and treated them as red, green and blue. Every ratio it produced was fiction.
Redone properly, painting each colour onto a canvas and reading the pixel back so the colour space is handled for me: 355 text nodes, 7 failures. Six of the seven are the same Tailwind token, gray-500, landing at 4.17 against the 4.5 floor for body text. The seventh is a badge, white on purple, at 3.96. That is one decision, not a redesign.
What bothers me is not the site. The entire difference between 51 and 7 was in my measuring instrument, and if I had not been suspicious of a number that big I would have shipped it as a finding. We are building a tool whose whole job is to tell people what is broken on their site. If the instrument is wrong they have no way to know, because checking it themselves is the thing they came to us to avoid.
So a genuinely open question. How do you test a checker? Not what it checks. How do you prove the check itself is not quietly lying to you?
Replies
Dial
The canvas-pixel-readback trick is a good instinct, it forces the browser's own color engine to do the color-space conversion instead of trusting your string parser. For testing the checker itself, the thing that's worked for us is building a small fixture set of hand-picked color pairs where we compute the "correct" ratio by hand or with a second, completely independent library, then assert the checker matches within some tolerance. Any time the checker's number and the reference number disagree, that's a bug regardless of which one is "right" until you've dug in. The other half is a regression trap: snapshot the checker's output on a real page you already trust, and fail loudly if a refactor changes the count without a corresponding code change to the actual rule logic. That's basically what caught your 51-vs-7 case in hindsight, a number that moved for no reason you could point to in the site itself.