Model routing
Also called LLM routing, model cascade.
Sending each AI request to the model best suited to it, instead of using one model for everything.
Last updated
What it is.
A router looks at a request and picks a model. The rule can be simple, such as task type or user plan, or it can be a small classifier. A common pattern is a cascade: try a smaller, cheaper model first and escalate to a larger one only when the answer fails a check.
Routing can also be about providers, not just model size: sending traffic to a backup when one provider is down, or to a provider in a specific region for data rules.
Common mistakes.
- 01
Routing without measuring quality on each path. Savings that lower answer quality are not savings.
- 02
Building a router more complex than the problem. A few clear rules often beat a learned router.
- 03
Forgetting that each model needs its own prompt tuning and structured output checks.
Work it out with a tool.
Related terms.
- Inference (AI)Running a trained AI model on new input to get an output, such as an answer, a label or an embedding.
- LLM evals (evaluations)Repeatable tests that score how well a model, prompt or AI feature does a specific task.
- Token (LLM)The unit an AI language model reads and writes, and the unit most LLM APIs bill by.
- LLM (large language model)An AI model trained on very large amounts of text that predicts the next piece of text, which lets it read, write and reason in language.
Guides and pages
Browse the full glossary or read answers to common questions.
