Miki Makhlevitch

I've spent the last year building writing tools, and the thing that surprised me most is where the difficulty actually sits.

Long-form is easy for a model. You give it a topic and it fills a page. Nobody's day is blocked waiting on a page.

The two-line reply is the hard one. To get it right you need to know what you're replying to, what was said three messages ago, what you already promised this person, and how blunt you're allowed to be with them. That context is scattered across a thread, a file, an old email, and your memory. So the ninety-second reply takes six minutes, and you do that forty times a day.

I recently built a benchmark for an AI retrieval system. The most revealing cases weren t the ones where the right answer ranked second instead of first. They were the ones where no relevant context existed at all.

A system that confidently retrieves something vaguely related can look functional while being completely wrong. So I added absence cases where the correct result is nothing.

It changed how I judge retrieval quality. Finding context matters, but knowing when there isn t any might matter more.

If you re testing an AI feature, do your evals include cases where the correct answer is I don t know ?

While I was starting distribution of my first saas, one thing which I overestimated was that the sales would happen very quickly as the product I made was good enough for it. Turns out, the product did not even had a market fit. I also ran ads with a very small budget and thought it will easily be gained within a few days, and I was very wrong here.

AI also does the same thing. If you ask it to expect some numbers based on what people experience for your saas when you distribute with some method X, it will show some very aspirational numbers. But, when ask it to be honest or practical or brutally honest for the same, it's answer changes (in some cases, significantly).