Attention as a budget

How does a model work out which earlier word a pronoun refers to?

Take the sentence "The trophy did not fit in the suitcase because it was too large." Every English speaker resolves it to the trophy without effort. Swap "large" for "small" and the same pronoun now means the suitcase. Nothing about the pronoun changed; nothing about the grammar changed. Only a fact about the world moved.

So a model that handles this cannot be following a syntactic rule about nearest noun. It has to be weighing candidates. Which raises the question worth chasing: when a model reads it, what does it actually do with the words behind it — and what stops it simply mixing them all together into mush?

Suppose you were designing this. At the position of it, you have representations of all the earlier words available. You want to pull in information from the one that matters. The blunt option is to average them all, but an average of "trophy", "suitcase", "fit" and "because" tells you nothing — averaging destroys exactly the distinction you need.

So you want a weighted average, where the weights are computed rather than fixed. That is attention, and the mechanism has three parts worth naming. Each position emits a query — roughly, "what am I looking for?" Each position also emits a key — "what am I, to someone looking?" The model scores every query against every key with a dot product, and where a query and key point in similar directions, the score is high. Then each position offers a value, the content it will contribute if attended to, and the output at it is the sum of all values weighted by those scores.

Now the constraint that gives this piece its name. Those raw scores could be anything — large, negative, wildly different in scale. They are passed through a softmax, which exponentiates them and divides by their total. The consequence is precise and important: the weights are all positive and they sum to exactly one. Every position spends the same fixed budget of attention, distributed differently.

Ask what that buys us. Because the budget is fixed, attention is inherently competitive. There is no setting in which every earlier word is attended to strongly; giving more weight to the trophy necessarily means giving less to everything else. That is why the mechanism can select rather than merely accumulate. A model with an unnormalised sum could satisfy "attend to the trophy" by simply turning up the trophy without turning anything down, and the representation would drift in magnitude as sentences got longer.

Where does the world knowledge come in? The query emitted at it is not a fixed "find me a noun" probe. It is computed from the representation at that position, which by the middle layers already carries information gathered from the surrounding words — including "too large". The keys at "trophy" and "suitcase" carry what those words are, again in context. So the compatibility score between them can encode something like the thing that would be too large in a failed fit rather than mere nounhood. It is not looking up a fact about trophies; it is that the training data made these directions align.

One honest caveat. It is tempting to say "there is a coreference head that does this", and researchers have found heads whose attention pattern aligns strikingly with syntactic and coreference relations — Clark and colleagues showed this for BERT in 2019. But the clean story oversells it. The work is spread across many heads and layers, heads that look interpretable on one distribution behave differently on another, and no one has a reliable method for pointing at a head and saying "that is where this pronoun was resolved". The mechanism is well understood; the attribution of any particular resolution to any particular head is not.

Three refinements.

First, this happens many times over in parallel. A layer runs several attention heads at once, each with its own queries, keys and values and therefore its own separate budget. One may track syntactic dependency, another positional proximity, another something with no human name. Their outputs are concatenated and mixed, so the "one fixed budget" picture is per head, not per model.

Second, the softmax's fixed total has a curious consequence: what does a head do when nothing in the sentence is relevant to it? It cannot abstain — the weights must still sum to one. In practice, models dump this unwanted mass on semantically empty positions, most often the very first token, a phenomenon documented as an attention sink. It is a real artefact with practical consequences: systems that trim old context must be careful not to evict the sink, or generation degrades.

Third, correct a natural misreading. A high attention weight is not the model "deciding" the pronoun's referent. It is one contribution to a representation that further layers will keep transforming. Attention weights are suggestive evidence about information flow, not an explanation of the output — which is exactly why interpretability work treats them with more caution than their intuitive appeal invites.

• Why the fixed budget also imposes a cost that grows with the square of the sequence length.
• How Winograd-style sentences became a test of world knowledge rather than of grammar.

Every term the collection defines is gathered in the glossary.