Evaluation suites

Why can you not tell whether a change improved an AI system by simply trying it?

You change a prompt. You run it on the example that was bothering you. The output is better. You ship.

Nothing about that sequence feels wrong — it is exactly how we debug ordinary software, and there it works. So I want to ask what is different here. In a normal program, the failing input either fails or it does not, and after the fix it does not, and you are done. What has changed about the system that makes the same procedure stop being evidence?

Start with the most uncomfortable property. Run the same prompt twice through a modern serving stack and you may not get the same answer. Some of that is deliberate — sampling at a non-zero temperature is random on purpose. But even with greedy decoding, outputs can differ between runs, because the arithmetic on a GPU is done in an order that depends on how requests happened to be grouped into a batch, and floating-point addition is not associative. Change the batch composition, change the last bits, occasionally change the chosen token, and from there the whole continuation.

So before you have changed anything at all, your single observation already has noise in it. What does that do to a before-and-after comparison of one example?

It destroys it, and in a specific way worth naming. You are trying to detect a small effect using a sample of one, drawn from a distribution you have not characterised. If the system gets that example right sixty per cent of the time, then a "before" failure and an "after" success is an event with probability around a quarter even if your change did nothing whatsoever. You have not measured an improvement; you have observed a coin.

Now the second problem, which is worse because it is silent. Suppose the change genuinely does help that example. What did it do to the other cases — the ones you did not look at? In ordinary code, the blast radius of an edit is bounded by what the edit touches. A prompt has no such structure: every instruction interacts with every input. Telling the model to "always cite the source document" fixes the ungrounded answers and, quite plausibly, breaks the cases where no source exists and it now invents one. You would never see it, because you were looking at your example.

That is the shape of the thing: high variance in each measurement, and unbounded coupling between behaviours. Which suggests what remedy?

Two, and they are different. Against the variance, you need repetition — the same case run several times, so you are comparing rates rather than outcomes. Against the coupling, you need breadth — many cases, chosen to cover the behaviours you care about, including the ones you are not currently worried about. A suite is what you get when you insist on both at once, and then insist on running it unchanged for every candidate change so the comparisons are like-for-like.

One honest caveat: nothing here promises a suite is right. A suite is a sample of the world, and the world moves. It tells you whether you got better on the cases you thought to write down — which is enormously more than one example tells you, and considerably less than "better".

It helps to see that a suite is doing two different jobs that people run together.

The first is regression detection: did anything that used to work stop working? This is the job most like ordinary testing, and it is where the value is highest and the argument least contested. It wants breadth and stability far more than it wants sophistication — a large set of ordinary, boring cases with cheap automatic checks catches more real damage than a small set of hard ones.

The second is measuring quality, and that is genuinely hard, because for most interesting outputs there is no single correct string. So the checks come in tiers: exact match or a schema validation where the task has a right answer; a programmatic property check where it has a rule but not an answer ("cites a real document ID", "returns valid JSON", "never names a competitor"); a model grading the output where nothing else will do. Each tier down is cheaper to write and less trustworthy, and the model-graded tier has its own bias that must itself be checked against human labels on a sample.

Two misconceptions to correct gently. First, a suite is not primarily about a headline score — the number is nearly meaningless in isolation, and useful almost entirely as a difference between two versions run the same way. Second, more cases is not automatically better: a suite so slow that people stop running it before shipping has a measured value of zero, which is why teams usually keep a fast subset for every change and the full set for releases.

And the failure to watch for is the one you cause yourself. Once a suite becomes the thing you optimise against, you will, over many iterations, fit to it — your changes start passing the suite without generalising. The standard defence is to hold back cases the tuning process never sees, and to refresh the set with real production failures as they arrive.

• How many cases you actually need, and how a confidence interval on a pass rate shrinks with sample size.
• Model-graded evaluation: where a judge model is reliable, where it is systematically biased, and how to calibrate it against human labels.
• The relationship between an eval suite and an A/B test in production — what each can see that the other cannot.

Every term the collection defines is gathered in the glossary.