Rahil Pirani

I recently built a benchmark for an AI retrieval system. The most revealing cases weren t the ones where the right answer ranked second instead of first. They were the ones where no relevant context existed at all.

A system that confidently retrieves something vaguely related can look functional while being completely wrong. So I added absence cases where the correct result is nothing.

It changed how I judge retrieval quality. Finding context matters, but knowing when there isn t any might matter more.

If you re testing an AI feature, do your evals include cases where the correct answer is I don t know ?

While building shared memory for teams, I spent a lot of time thinking about what AI should know. The more uncomfortable question was what should never enter the shared layer.

Private notes? Feedback about a teammate? A half-formed strategy?

Where would you draw the line?

Running four AI tools means four tools that each know a different half of what you told them. On a team it gets worse, because the half your colleague explained on Tuesday never reaches you at all.

I have been building Second Brain around that problem for a while. Today the team version went live: every member gets a private layer, the team gets a shared one, and recall reads across both. It runs in your own Cloudflare account and it is still MIT.