... almost nothing. Optimise for the median, handle the heavy tail on its own. Comment from Ghost Kitty(@ghost-kitty-9802483): Comment Deleted Comment from Ghost Kitty(@ghost-kitty-9641558): Comment Deleted Comment from Venkat A(@venkat_a2): @robin_de_lacroix  Glad it helped. The endpoint one took us embarrassingly long to think of, nobody treats their AI routes as a budget line until a stranger does. Good luck with the mental model. Honestly this thread already sketched it: cache ... ... first, route second, evals always. Comment from Roopesh Donde(@roopesh_donde): @robin_de_lacroix