One improbable word

Why can a single unlikely word early in an answer derail everything that follows it?

Ask a language model the same question twice and you will sometimes get one answer that is careful and correct, and another that is confidently wrong from the third sentence onward. Nothing changed: same model, same weights, same prompt. If the machinery is identical, where did the difference come from?

The tempting answer is that the model "knew less" the second time, or was "in a bad mood", which is not a mechanism at all. Let us look instead at what actually differs between two runs, and see whether that alone is enough to explain a whole answer going sideways.

Start with what the model does at each step. It does not produce an answer; it produces a probability distribution over the next token, given everything in the context so far. Something outside the model then chooses one token from that distribution — and that choice is the only thing that differs between two runs at nonzero temperature. Under nucleus sampling, for instance, the sampler keeps the smallest set of tokens whose probabilities sum to some threshold and draws from within it, so a token sitting at two percent is not excluded. It is unlikely, not impossible.

Now ask the question that matters: what happens to that token afterwards? It is appended to the context. And at the next step the model conditions on the whole context, with no marker distinguishing "text I was given" from "text I happened to sample". A word the model considered improbable a moment ago has become, from the next step's point of view, simply part of what is true so far.

Notice what that does to the odds. Before, the improbable claim had a two percent chance. After, its probability is one — it is in the record. The model's task changes from is this so? to given that this was said, what follows? And what the model is extremely good at, because it is what training rewarded, is producing text that is locally coherent with its own preceding text. So it does the coherent thing: it supports the claim. It supplies a reason, a date, a citation, a consequence. Each of those, once emitted, is itself context for the next step.

Do you see the loop close? A low-probability token raises the probability of tokens that fit it; those tokens are emitted; they raise it further. This is what people mean by compounding: the error is not merely uncorrected, it is actively reinforced by the same mechanism that makes fluent text fluent. Researchers call the visible version of this hallucination snowballing — a model committing to a wrong premise and then generating a tidy scaffolding of support for it.

There is a second reason, quieter but real. Training is done with teacher forcing: at every step the model is shown the true prefix and asked for the next token. It therefore never practises the situation it faces constantly at inference — continuing from a prefix it wrote itself, containing its own mistake. That mismatch has a name, exposure bias, and it is one honest explanation for why recovery is weaker than you would hope. How large its contribution is, compared with plain coherence pressure, is not settled; both are real and they are hard to separate experimentally.

One more observation before we generalize. Position matters. A wrong token in the last clause has almost nothing left to contaminate. The same token in the first sentence sits in the conditioning set of every token afterwards. The damage is roughly proportional to how much text still has to be written under it — which is why an early framing sentence, or a first step in a chain of reasoning, is far more load-bearing than its length suggests.

Two corrections are worth making, because the picture above is easy to over-read.

First, this is not a claim that a single unlikely token always derails an answer. Most improbable tokens are harmless — a slightly odd adjective, an unusual connective. What matters is whether the token is load-bearing: whether later text has to be consistent with it. "Furthermore" carries no commitments. A wrong middle name, a wrong sign, a wrong first step in an arithmetic chain carries many.

Second, setting temperature to zero does not solve it. Greedy decoding removes the randomness, not the compounding. If the model's highest-probability token happens to be wrong, the same reinforcement runs, and it runs identically every time — which is arguably worse, because the failure is now reproducible rather than rare. The mitigations that do help all work by breaking the single-chain conditioning: sampling several independent answers and comparing them, letting the model critique a completed draft in a fresh pass, or grounding claims in retrieved text so the context contains something the model did not write.

That last point is the general shape of it. The problem is not that the model is uncertain — uncertainty is fine and even useful. The problem is that the architecture provides no way for uncertainty to persist. A token is either in the context or it is not; there is no "probably" in the record. Every mitigation is essentially an attempt to keep some doubt alive outside the token stream.

• Why self-consistency methods — sampling several answers and taking the majority — catch some errors but systematically miss others.
• Whether a model's own reported confidence at the moment of sampling is any guide to whether a token will turn out to be load-bearing.
• How retrieval changes the picture by putting text into the context that the model did not generate.

Every term the collection defines is gathered in the glossary.