The most useful AI eval I wrote expected no answer

I recently built a benchmark for an AI retrieval system. The most revealing cases weren’t the ones where the right answer ranked second instead of first. They were the ones where no relevant context existed at all.

A system that confidently retrieves something vaguely related can look functional while being completely wrong. So I added absence cases where the correct result is nothing.

It changed how I judge retrieval quality. Finding context matters, but knowing when there isn’t any might matter more.

If you’re testing an AI feature, do your evals include cases where the correct answer is “I don’t know”?