Evaluating retrieval separately

When an answer built from sources is wrong, how do you tell whether the finding or the writing failed?

A system that searches a corpus and then writes an answer from what it found has exactly one output a user sees: the answer. So the natural way to measure it is to score that answer — against a reference, or by a human, or by another model acting as judge. One number, end to end, honest about what the user experiences.

The trouble appears the moment the number goes down. The answer was wrong. Which half broke? And notice that the end-to-end score, whatever it is, cannot tell you: it is a single scalar summarising a pipeline with two very different failure modes, and every fix you might attempt — change the chunking, change the model, change the prompt, change the index — is a fix to one half or the other. The measurement you have does not name a suspect.

Let us be precise about how many ways it can fail. The system does two things: it selects evidence, and it writes from that evidence. So there are four combinations, and it is worth asking whether all four are distinguishable from the outside.

Right evidence and right writing gives a good answer; wrong and wrong gives a bad one. Both easy. Now the interesting pair. The evidence was wrong but the writing faithfully used it — a fluent, well-grounded answer based on the wrong passage. From the outside that looks exactly like a hallucination, and teams respond by blaming the model, tightening the prompt, lowering the temperature. None of it helps, because the model did its job correctly on the input it was given.

And the mirror case: the evidence was right and the writing ignored it, contradicted it, or leant on the model's own prior knowledge. That looks like a retrieval failure to anyone who assumes a grounded system uses what it is given.

So two of the four cases are systematically misattributed by an end-to-end score, and the two are misattributed in opposite directions. Which means the end-to-end number does not merely lack detail — it actively points at the wrong component roughly half the time you are debugging.

What would separate them? For the retrieval half, you need to know whether the passage containing the answer was in the retrieved set at all — answerable without running the writer. Mark which passages actually contain the answer for a set of questions, then measure how often they appear in the top k. Recall at k is the ceiling on everything downstream, and ordering measures like nDCG tell you additionally whether the right passage arrived near the top, which matters when the writer reads only the first few.

For the writing half the key move is different: judge the answer against the retrieved passages, not against the truth. Is every claim supported by the text supplied? That is faithfulness, and notice its useful property — being defined relative to the input the writer actually received, it stays valid even when retrieval was garbage. A faithful answer to bad evidence scores well here and badly end to end, which is exactly the discrimination we were missing.

Does that fully separate the two? Not quite. A third component hides between them: a retriever can score well on recall at ten while burying the answer among nine distractors, and a writer that then goes astray is failing on a genuinely harder input. So careful setups also measure context relevance — what fraction of the supplied context was useful — which attributes the "too much noise" case to retrieval, where it belongs, rather than to the writer who drowned in it.

One more consequence, and it is the practical payoff. Retrieval evaluation needs no generation at all: it is a fast, deterministic sweep over an index that can run on every commit. Generation evaluation is slow, costs model calls, and carries its own error rate. Merging them into one score does not just confuse attribution; it drags the cheap loop down to the speed of the expensive one.

Three refinements.

First, this is component testing, with component testing's classic weakness: both components can pass while the system fails, because the fault lives in the interface. Keep the end-to-end score. Its job is detection, not attribution.

Second, the labels are the expensive part and the part most often skipped. Recall at k requires knowing which passages truly answer each question, which means human judgement over a question set drawn from your own traffic — far more useful than a public benchmark on someone else's corpus.

Third, on model-as-judge scoring: it is the standard way to measure faithfulness at scale and it is genuinely useful, but its agreement with human raters varies by task and prompt, and it carries known biases toward longer answers and toward text from similar models. Calibrate against a human-labelled sample before trusting a movement of a few points.

• How to build a labelled question set from real traffic without it drifting away from what users now ask.
• Whether context precision or context recall is the better early warning when a corpus grows.

Every term the collection defines is gathered in the glossary.