Two-stage retrieval

Why fetch fifty candidates only to throw away forty-five of them with a slower method?

There is something that looks like waste in a modern search stack. The first stage pulls fifty or a hundred documents. The second stage then runs a considerably more expensive model over every one of them, reorders them, and keeps five. Forty-five results were fetched and scored only to be discarded.

The obvious objection writes itself: if the second model is the better judge, why not just use it on everything and skip the first stage? And if the first model is good enough to shortlist, why not trust its top five? Either way, one of the two stages looks redundant. Working out why both are needed turns out to be a lesson about what makes a scoring function cheap.

Begin with the cost question, because it is the one with a hard answer. Ask what a retrieval system must do to score a query against ten million documents in fifty milliseconds. It cannot look at ten million documents. So the only way this is possible is if most of the work was done before the query arrived — which forces a specific shape on the scorer: each document must be reducible to something computed in advance, independent of any query.

That is exactly what a dense embedding model does. It reads a document alone and produces a vector; it reads the query alone and produces another; relevance is the similarity between them. Because the document side never sees the query, all of it can be precomputed into an index supporting approximate nearest-neighbour lookup, so the query touches a tiny fraction of the corpus. Lexical scoring works on the same principle — an inverted index is a precomputed structure too.

Now ask what that independence costs. If the document vector is fixed before the query exists, it must summarise the document for all possible queries at once. A single vector of a few hundred numbers cannot preserve everything; it preserves the gist. So the fine distinctions — which of these two passages actually answers the question rather than merely discussing its subject, whether the negation matters, whether the entity in the query is the same entity as the one in the passage — are precisely the distinctions that get averaged away.

What would a scorer that keeps them look like? It would have to read query and document together, letting each word of the query attend to each word of the document. That is a cross-encoder: one model, one input consisting of both texts, one relevance score out. It is much more accurate on exactly the fine distinctions the first stage loses.

And what does it cost? Notice you cannot precompute anything, because the score depends on the pair. So the work is one full model pass per document per query. Over ten million documents that is not slow, it is impossible — and there is no index that helps, because there is no query-independent quantity to index.

Which resolves the apparent waste. The two stages are not two attempts at the same job; they are two scorers with opposite cost structures, and neither can do the other's work. The first is cheap because it is query-independent, and inaccurate for the same reason. The second is accurate because it is query-dependent, and unaffordable for the same reason. Putting them in series lets the expensive one be spent only where it can change the outcome.

So how many candidates should the first stage pass on? Here the reasoning becomes concrete. The second stage can only reorder what it is given; a document the first stage missed can never be recovered. So the first stage's job is not precision at all — it is recall at whatever depth you are willing to pay to rerank. If the right document is in the top hundred ninety-five percent of the time and in the top five only sixty percent of the time, then a reranker over a hundred candidates has a ceiling of ninety-five, and the forty-five discarded results were the price of that ceiling. The discards are not waste; they are the insurance premium.

That also tells you when the second stage is pointless. If the first stage's recall at 100 is barely better than its precision at 5 — a small or homogeneous corpus, say — there is nothing for the reranker to promote, and you have bought latency for nothing.

Three clarifications.

First, "two stages" is the common case, not a law. Web-scale systems run several, and the principle generalises: each stage should be roughly an order of magnitude more expensive and see an order of magnitude fewer items.

Second, the reranker is not simply "a better model." It is a different architecture with a different input contract. Swapping in a larger embedding model does not give you a reranker's abilities, because it still never sees query and document together.

Third, on numbers: the accuracy gain from cross-encoder reranking is consistently reported on standard benchmarks, but its size depends heavily on corpus and query type, and the latency cost is real. Measure recall at your chosen candidate depth on your own data before assuming either the benefit or the price.

• How late-interaction models, which keep per-token document vectors and defer some interaction to query time, sit between the two stages.
• Whether a reranker should score for topical relevance or for answer-bearing usefulness, which are not the same target.

Every term the collection defines is gathered in the glossary.