How do you decide when a cheaper AI model is good enough?

Within one provider, the cheapest and most expensive model tiers can be about 20x apart per token. So once one AI feature is most of the bill, moving it down a tier looks like free money.

The saving is easy to work out: price the feature’s real token counts on both models and subtract. What you give up is harder to see. Overall leaderboards average across very different tasks, and a model can rank well overall and still drop sharply on the one thing your feature does.

Here’s an example. A code-review feature would save about $44 a month on a cheaper model, but its coding benchmark score drops from 73 to 52. For most products that’s a bad trade, because one customer who leaves over sloppy reviews costs more than the saving.

The rough rule I’ve landed on:

• Switch when the quality gap on that kind of work is small.

• Test first when it’s close. Send 5 to 10% of the feature’s traffic to the cheaper model for a normal week, and compare retries, edits and complaints.

• Keep paying when the gap is large.

The part I haven’t solved is the “close” zone. Benchmarks aren’t your workload, and a week of traffic doesn’t always settle it. How do you decide? Proper evals, gut feel, or ship it and watch the complaints?