FastRouter.ai - Route requests to the right LLM for cost, latency & quality
by•
FastRouter is a unified AI gateway and control plane for developers and enterprise teams building with LLMs. It routes every request to the right model across 200+ LLMs through one OpenAI-compatible API, optimizing for cost, latency, quality, and reliability. With intelligent routing, failover, observability, and governance, teams can scale AI apps without vendor lock-in or code changes.
Replies
love the idea of a single endpoint for all 200+ models. makes testing new releases so much cleaner.
@charles_eric3 Thanks, Charles! Exactly. Trying a new model shouldn’t mean integrating another SDK. A single API makes it much easier to compare models and find the right fit for your app :)
am sending this straight to our engineering lead we desperately need something like this right now.
@irsa_doham Thanks for passing it along, Irsa! 🙌 Would love to hear what your team is working on. Happy to answer any questions your engineering lead has or help them get started! Do send us a note on support@fastrouter.ai.
Routing across 200 plus models through one OpenAI compatible endpoint is the kind of thing teams only appreciate once they have written the same retry and fallback logic by hand. Optimizing for cost, latency and quality together is the interesting part, since those usually pull against each other. How transparent is the routing decision when you want to debug why a request went a certain way?
@karimbenkeroum Thank you for a great question. The short answer is that it depends on the routing mode.
1. Explicit policies via Virtual Model Aliases are fully transparent. The Activity Log shows the alias, strategy, every attempt including failovers, final model, and per-attempt cost/latency.
2. Auto-Router is partially transparent today: you see the task category the request was classified into, the model chosen, and the metrics, but not the score breakdown behind the pick.
3. AI Insights / Custom AI Evals is fully open: scores, LLM-judge reasoning per request, and side-by-side responses behind every recommendation.
Happy to clarify any of these or do a detailed demo to explain all of them.
congrats on the launch. my biggest headache is latency and failover at the same time, since we work with voice. once audio starts playing you can't quietly retry, so the fallback has to be decided before the first byte goes out. does your routing optimize for time to first token and p95 rather than the average? and if a provider dies mid stream, do you restart on another upstream or just surface the error?
@danivs10 Great questions.
1. While the order of model x provider choices to be made take into account cost and error rates, the decision to failover for a particular request is based on a combination of fixed values (e.g. when exactly to timeout) to upfront provider errors that we receive.
2. If the stream has already started and the provider errors out midway, we don't retry. We surface the error on the stream and end it.
Happy to discuss / add a feature if useful for your use case - e.g. build a configurable rule for timeout, retry, etc or anything else that may be valuable. Do reach out to us on support@fastrouter.ai.
Cost, latency and quality is three knobs where the third one does all the work and is the hardest to measure. We route across 30+ models in our own product and the thing that bit us wasn't a slow call or an expensive one, it was a small model returning something clean, confident and wrong, which passes every format check you have. So the number I'd want out of a control plane is cost per accepted output, not cost per call, because a model that's 4x cheaper and needs two regenerations just moved the bill somewhere nobody is measuring. If the quality signal behind routing is benchmarks rather than what your own users kept, it'll pick the cheap model at exactly the wrong moments.