Model routing
A Socratic walk-through of model routing — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why send some questions to a weaker system on purpose?
If you have access to the strongest model available, sending anything to a weaker one looks like deliberately choosing a worse answer. And for a single question it is — or at least it risks being.
But hold the question up differently. You are not answering one question; you are answering a million of them, under a budget and a latency target. Does the strongest model still look obviously right? Only if every one of those million questions is hard. So the first thing to test is whether that is true.
Reasoning it through
REASONING #It is not. Look at what real traffic contains: classify this ticket, extract these three fields, rewrite this sentence politely, summarise a paragraph, answer a factual question in one line. Then, in a much smaller share, the things capability is for — multi-step reasoning, subtle code, ambiguous instructions, long documents held in mind at once. The distribution is lopsided, and the price gap between tiers is not a few percent; between a small model and a frontier one it is commonly an order of magnitude or more per token, with a similar gap in latency.
So the shape of the opportunity is clear: pay for capability only where capability is used. The difficulty is that you must decide which question is which, and you must decide it cheaply, and — this is the part that matters — usually before you have seen the answer.
That last constraint splits the field in two.
The first family predicts. A router looks at the incoming request and estimates whether the cheap model will do; if not, it sends the request up a tier. This is what RouteLLM (from LMSYS, 2024) does, training routers on human preference data to predict which queries need the stronger model. It is fast, because the router is small, and it never pays twice. But ask yourself what it is doing: judging difficulty from the question alone, which is a genuinely hard prediction. Some questions look trivial and are not.
The second family tries, then checks. Run the cheap model first, evaluate its output, and escalate only when the output fails — the cascade approach set out in FrugalGPT (Chen, Zaharia and Zou, 2023). This has a real advantage: it decides using the answer, which is far more evidence than the question. The cost is that an escalated request pays for both calls plus the wait, so a cascade only pays off when the cheap model succeeds often.
Which raises the question that decides whether a cascade works at all: how do you tell that the cheap answer was good? Asking a model how confident it is turns out to be a weak signal — self-reported confidence from language models is poorly calibrated, and improving that calibration is an open research problem. The signals that hold up are the ones outside the model: does the generated code compile and pass the tests, does the JSON validate against the schema, does the SQL execute, does the extracted field appear verbatim in the source document, did the tool call return an error. Where a task has a checkable output, cascading is on firm ground. Where it produces prose that can only be judged by another model, it is much shakier.
Two more constraints, both easy to forget. The router's own cost is subtracted from every saving — a router that needs a large model to make its decision has spent the money it was trying to save. And the two errors are not symmetric: sending an easy question to the expensive model wastes a fraction of a cent, while sending a hard one to the weak model produces a confidently wrong answer that someone may act on. That asymmetry means the threshold should not be set to maximise average accuracy per unit cost; it should be set from what a bad answer costs in this application, which is a product decision more than an engineering one.
Finally, cost is the most common reason to route, not the only one. Requests may go to a local model because the data must not leave the building, to another because the context is too long for the cheap one, or to a specific provider for a jurisdictional reason. Same mechanism, different objective.
The analogy
THE ANALOGY #Think of triage at a clinic. Not everyone who walks in needs the consultant. A nurse takes a quick look, and most people are dealt with there and then; a few are sent up. The consultant's time is preserved for the cases that need it, and the clinic sees far more patients than it otherwise could.
The triage nurse examines the patient before deciding — whereas a predictive router must judge from the request alone, before any answer exists, so it is guessing at a difficulty it has no way to observe yet; a cascade is closer to the nurse, and pays for the extra look.
Clarifying the model
THE MODEL #Three refinements.
First, "weaker" is doing less work than it seems. Small models are not uniformly worse; they are worse on the things scale buys — long-horizon reasoning, obscure knowledge, following intricate instructions. On a narrow, well-specified task with a clear output format, the gap can be small or absent, and a fine-tuned small model can beat a general large one on its own task. Routing exploits that unevenness; it is not simply buying less quality.
Second, a router is a model of your traffic, and traffic moves. Users change what they ask, prompts get rewritten, model versions change underneath. A router validated once decays quietly, and the decay is invisible unless you measure it — the useful instruments are the escalation rate over time and a periodic sample of cheap-tier answers re-judged against the strong model.
Third, routing is not the same as degrading under load. Load shedding says "you get a worse answer because we are busy". Routing says "this question does not require the expensive path". If you cannot say which questions those are, you do not have a router — you have a random discount on quality.
A picture of it
THE PICTURE #How to readRead left to right; band thickness is volume, and the numbers are illustrative proportions rather than measurements of any real system. The split at the left is the router's decision, made before any answer exists. Follow the small-model band: most of it reaches the accepted answers directly, and that thick band is the entire saving. The thin band peeling off to escalation is the cascade at work — those requests pay twice, so widening it eats the saving from the other. The design is the ratio between those two bands.
What became clearer
WHAT CLEARED #Routing is not a decision to accept worse answers; it is the observation that difficulty is unevenly distributed and price is not, so capability should be spent where it is used. Everything hard about it collapses into one question — how do you know which requests are which — and the honest answers are either a prediction from the question alone or a cheap attempt plus a checkable test on the output, with the second being far sturdier wherever the output can actually be checked.
Where to go next
ONWARD #- Calibration: why a model's stated confidence is a poor escalation signal, and what substitutes for it.
- Fine-tuning a small model for one task, and when that removes the need to route at all.
- Prompt caching and batching, which reduce the same bill by a different route and compose with this one.
Key terms
TERMS #| Term | What it means |
|---|---|
| Router | a component that predicts, from the request, which model tier should handle it. |
| Cascade | running a cheaper model first and escalating to a stronger one only when its output fails a check. |
| Escalation rate | the share of requests that reach the expensive tier after starting cheap; the number that says whether a cascade is paying. |
| Calibration | how well a model's expressed confidence matches its actual accuracy. |
| Cost tiering | offering several capability and price levels and assigning work across them. |
Every term the collection defines is gathered in the glossary.