Lost in the middle

Why does a model use the beginning and end of a long document more reliably than its middle?

Suppose you paste thirty documents into a model's context and ask a question that exactly one of them answers. The model can hold all thirty comfortably — you are well inside the window. Would you expect the answer to depend on which of the thirty happens to contain the fact?

Intuition says no. Attention, as usually described, compares the query against every position; there is no queue, no scrolling, no fatigue. And yet when Liu and colleagues measured this carefully in 2023, published in TACL the following year under the name that stuck, accuracy formed a shallow U: highest when the needed document sat first or last, lowest when it sat in the middle. In their harder settings, a model with the fact buried mid-context did no better than the same model given no documents at all.

So which assumption was wrong — that everything in the window is equally available, or something quieter underneath it?

Begin by noticing what the U shape rules out. If the model simply degraded with distance — attending less to what is further from the question — you would see a slope, not a U. A curve that rises again at both ends says at least two separate advantages are in play, one favouring the start and one favouring the end.

The end is the easier of the two. The next token is generated at the far right of the sequence, and everything near it is close in position — and the relative position encodings in wide use, such as RoPE, are known to weaken the interaction between tokens that are far apart. Recency is not a bias the designers inserted; it falls out of how position is represented.

The beginning is stranger, and it has a good candidate explanation. Xiao and colleagues, in 2023, described what they named attention sinks: the very first tokens of a sequence soak up a large share of attention weight across layers, apparently because softmax forces the attention weights to sum to one, so a head with nothing it particularly wants must still put its mass somewhere — and the earliest positions, visible to every later token, become the default dumping ground. Whatever the fuller story, the opening of a sequence is structurally privileged.

Then add the influence I would put the most weight on, because it needs no architecture at all. Ask what the training and fine-tuning data look like. Instructions come at the top of documents. Questions come at the end of them. Abstracts open papers, conclusions close them, and the middle of a long text is where the supporting detail lives. A model shaped by that distribution learns, quite sensibly, that the beginning and end of the input are where the load-bearing material usually sits. On this account the U is not a flaw in attention; it is a learned prior about how humans arrange documents, applied to a case where it does not hold.

Now the honest part. Which of these three dominates is not settled. They are compatible, they are hard to separate experimentally, and the published work argues for different weightings. Treat them as candidates, not as a mechanism.

There is a second honesty check, and it matters more for practice. This is a property of particular models and their training, not a law of transformers. Needle-in-a-haystack retrieval tests on more recent long-context models are often close to flat, because once the effect was named it became something labs explicitly train and evaluate against. The instructive pattern is that the more the task demands — several facts to combine rather than one string to find, or reasoning over them rather than quotation — the more the middle penalty tends to reappear. Retrieval got fixed faster than use.

One consequence to check yourself against: does it follow that a longer context is always better? It does not. Extending a window makes more material admissible; it does not make it equally influential, and Liu's group found that extended-context variants of models were not reliably better at using their extra room than the shorter originals. Padding a prompt with everything you have can lower accuracy even while every relevant fact is technically present.

Three clarifications. First, this is not about the window overflowing. Every document was inside the limit; the question is how influence was distributed within it, and capacity is a separate axis from use.

Second, "lost" is too strong. The middle is attended to less reliably, not erased — performance sags, it does not fall to zero, and it varies with model, prompt, and how demanding the task is.

Third, and worth stating because the design implication is real: position is a variable you control. If you are assembling a prompt from retrieved chunks, ordering is not cosmetic. Put the most decisive material at the top or the bottom, keep the middle for context you can afford to have underweighted, and rank your retrieval rather than dumping it. Where accuracy matters and the input is long, the more robust patterns are to narrow the input before the model sees it, or to process a long document in sections rather than in one pass.

• What needle-in-a-haystack benchmarks do and do not measure, and why multi-fact variants are harder.
• Attention sinks, and why keeping the first few tokens makes streaming attention work at all.
• Whether reranking retrieved passages by relevance beats simply retrieving fewer of them.

Every term the collection defines is gathered in the glossary.