Look at a month of prompts in almost any AI product and you'll find a long tail of simple requests: reformat this, summarise that, classify this ticket. Then there is a smaller set of hard ones: multi-step reasoning, ambiguous questions, long documents. Paying top-model prices for all of it is like sending every support call to your most senior engineer.
Two approaches
| Router | Cascade | |
|---|---|---|
| How it works | Predicts which model should answer, then calls one model | Tries a cheaper model first; escalates if the answer isn't judged reliable |
| Cost on hard requests | One call to the strong model | Several calls, including the failed cheap attempt |
| Latency | Predictable | Higher on escalated requests |
| Hardest part | Training a router that predicts difficulty well | Judging reliability of an answer |
FrugalGPT, from Stanford, is the best-known cascade: it queries models in sequence and stops when a response is judged reliable3. RouteLLM, from the LMSYS and Berkeley team behind Chatbot Arena, trains routers on human preference data to choose between a stronger and a weaker model per request1.
What the research showed
On public benchmarks, the RouteLLM project reported cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K compared with using only GPT-4, while achieving 95% of GPT-4's performance2. The spread is the lesson: savings depend heavily on how much of the workload is easy. Conversational traffic had lots of easy requests; maths problems had fewer.
Benchmark savings don't transfer. The only number that matters is the saving on your traffic at a quality level your users accept, measured on your own evaluation set.
How I design routing
- Start with rules, not a learned router. Route by task type the product already knows: classification and extraction to a small model, open-ended reasoning to a large one. This captures much of the saving with no model to maintain.
- Put routing in the gateway, not in application code, so policies change in one place. The same gateway records cost per call, as in FOCUS and AI spend.
- Evaluate every route. Each model on each route needs a passing score on the relevant slice of your evaluation set before it takes traffic.
- Add a learned router only when the rules plateau, and train it on your own data: requests labelled with which model's answer was acceptable.
- Monitor quality per route: user feedback, escalations and sampled reviews. Routing failures look like a gradual decline in satisfaction, not an outage.
- Keep sensitive routes fixed. Requests involving regulated data may be pinned to private models, as in private LLMs on-prem, regardless of cost.
Don't forget the cheaper levers
Routing is one of several cost tools. Prompt caching and shared prefixes, covered in LLM inference serving, shorter outputs and better retrieval that sends fewer tokens can each save as much as routing, with less risk. I usually apply those first, then route.
How I can help
I design AI gateways, routing policies and evaluation loops that cut model spend without quietly cutting quality. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1RouteLLM: Learning to Route LLMs with Preference DataOng et al. (arXiv) · 2024
- 2RouteLLM: an open-source framework for cost-effective LLM routingUC Berkeley Sky Computing Lab
- 3FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving PerformanceChen, Zaharia and Zou, Stanford (arXiv) · 2023
- 4What is LLM model routing?NeuralTrust