Speculative decoding

Why can a large model produce its text faster when a smaller, less capable model writes the first draft?

Here is an arrangement that sounds like it must be a trade. A small, weaker model writes several tokens ahead; the large model looks at them. The text comes out two or three times faster — and, the claim goes, it is the same text the large model would have produced alone.

Both halves are suspicious. A second model is more work, not less, and identical output from a pipeline containing a worse model sounds like something for nothing. So either a claim is loose, or generating text costs something quite different from what we assume.

Start with what producing one token costs. The machine must pass the current context through every weight in the model. Those weights live in memory, and the arithmetic cannot begin until they arrive. At batch size one, essentially every weight is read, used for a few operations, and discarded. The time is therefore set by how fast weights can be moved, not by how fast the processor can multiply — and accelerators multiply vastly faster than they fetch. The processor spends most of a token waiting.

Now the observation everything hinges on. Hand the model five candidate tokens instead of one and ask for its prediction at each position: that is one pass. The weights are read once, and the extra arithmetic rides along in capacity already idle. So scoring five tokens costs roughly what scoring one costs, while generating five costs five passes, because each token must exist before the next can be conditioned on it.

There is the asymmetry: generation is inherently serial, verification is parallel, so anything that proposes plausible tokens cheaply converts serial work into parallel work. Do you see why the proposer's quality is not the point? It only has to be right often enough, and cheap. So the draft model generates k tokens by its own serial passes, cheap because its weight set is small, and the large model does one pass over the drafted sequence to obtain its own distribution at every position. Then comes the part that makes the identical-output claim true rather than a slogan.

At each position, let p be the large model's probability for the drafted token and q the draft model's. Accept with probability the smaller of 1 and p/q. On rejection, sample a replacement not from p but from the normalised positive part of p minus q — the mass the large model wanted that the draft under-supplied — and stop, discarding the rest of the draft. Work through the algebra and the token finally emitted is distributed exactly according to p. That is a theorem about rejection sampling, not an empirical observation: a weaker draft does not degrade the text, it is rejected more often, which costs speed.

How much speed? Idealise by assuming each drafted token is accepted independently with probability a. From a draft of length k, the expected number confirmed before the first rejection, plus the one token the big model always emits itself, is 1 + a + a*² + … + *a*ᵏ, summing to (1 − *a⁽ᵏ⁺¹⁾)/(1 − a). With a = 0.8 and k = 4 that is (1 − 0.32768)/0.2 = 3.36 tokens per large-model pass instead of one. Put the draft model's cost at a twentieth of the large one's and the cost per accepted token becomes (1 + 4 × 0.05)/3.36 = 0.36 of what it was — a bit under threefold.

That is an idealisation, explicitly: acceptance is not independent across positions — a draft that has gone off the rails stays off it — and the k draft passes are serial, adding latency a batched cost model hides. I am also not quoting published speed-up figures, which depend on model pairing, sequence length, batch size and hardware, and move quickly.

Now the falsification test, which is unusually clean. The load-bearing claim is that the output distribution is preserved exactly. Run the same prompt thousands of times, once with ordinary sampling from the large model, once with speculative decoding using a deliberately poor draft model, and compare the token distributions. They must be statistically indistinguishable, the poor draft showing up only as a lower acceptance rate. The refuting observation would be a measurable shift in output distribution as the draft model changes — meaning the acceptance rule is wrong or wrongly implemented, and the technique is a quality trade after all.

The misconception to name is that this is a quality-versus-speed knob. It is not. Quantisation is a quality trade — it changes the model's numbers and its outputs. Speculative decoding changes only when computation happens: the accept-reject rule leaves the sampled distribution untouched, and the draft model's weakness is paid for in throughput, never in text. Both speed up inference, with a sharp fixed point of difference: one alters the model, the other the schedule.

The second clarification is durability. The technique is arbitrage on a gap between two hardware ratios — how fast weights can be fetched versus how fast arithmetic can be done — and it disappears when that gap closes, as it does under heavy batching: with many sequences served at once each weight fetch is amortised across all of them, the system becomes compute-bound, and the drafting and verification arithmetic stops being free. So the speed-up is real at interactive latency and shrinks as the server fills. That framing survives a change of models: the same trick appears wherever a step is serial to produce but parallel to check — a processor speculating past an unresolved branch, an optimistic transaction validated at commit.

It follows that the draft model is incidental. Any cheap source of plausible continuations works — an n-gram lookup over the prompt, extra prediction heads, a retrieved passage. The method needs not a small model but cheap proposals verifiable in parallel.

• Why a processor's branch predictor is the same bargain in different clothing, and what its misprediction penalty corresponds to here.
• How tree-structured drafts, proposing several alternative continuations at once, raise the accepted-token count per verification pass.

Every term the collection defines is gathered in the glossary.