Cheapest model that passes your eval is the wrong routing rule

Most routing I've seen picks the cheapest model that clears some benchmark, and that optimises the one cost that barely matters. Tokens are cheap. The expensive thing is an answer that looks right, ships, and gets caught by a person twenty minutes later, or never.

So the number I care about isn't cost per call, it's cost per accepted output. A model that's 4x cheaper but needs two regenerations and a careful human read is not cheaper, it just moved the bill somewhere you weren't measuring. Once you count it that way the routing table changes, and a few tasks that felt like obvious cheap-model work move back up.

The part I still don't have a good answer for is spotting wrongness at route time instead of after. Self-reported confidence is close to useless here. What's helped a bit is routing on task shape rather than prompt content, because some categories are reliably where things go quiet and wrong.

Curious what other people actually route on, and whether anyone is measuring acceptance rather than spend.