Chunking

Why does how you cut a document into pieces matter more than which search engine you use on them?

When a retrieval system disappoints, the first instinct is to change the engine: swap the embedding model, try a different vector database, turn on a fancier index. Those are the parts with names and benchmarks, so they feel like the parts that decide the outcome. Meanwhile the step that decided what the engine would ever be allowed to find — the decision to cut documents into pieces of a certain size at certain places — was made once, early, by whatever default the library shipped with.

That ordering is worth questioning. If the pieces are the unit of everything downstream, then no engine can retrieve a good passage that chunking never created. So what exactly does a cut decide?

Start with what a chunk is in the pipeline. It is three things at once, and this is the whole difficulty. It is the unit that gets embedded or indexed, so it determines what the match is computed over. It is the unit that gets scored and ranked, so it determines what "relevant" is measured against. And it is the unit that gets handed to the reader — a person or a language model — so it determines what evidence they actually have.

Ask whether one size can be right for all three. Consider a cut so small that each chunk is one sentence. The match is now beautifully sharp: an embedding of a single sentence is a vector about one thing, not an average of eight things, so a query about that thing lands squarely on it. But hand that sentence to whoever must answer, and it often says "This limit does not apply in that case" — with no way to know what limit or which case. Precise retrieval, unusable evidence.

Now cut the other way, one chunk per document. Every retrieved piece is self-contained. But the embedding is now the smeared average of a report covering twenty topics, so a query about any one of them matches it weakly and matches nothing else strongly; and the reader receives forty pages to find one clause in. Complete evidence, blunt retrieval.

So there is a genuine tension, not a tuning knob with a best setting: matching wants small pieces and answering wants large ones. Does anything break the tension rather than trade against it? Two things do, and both are instructive because they work by refusing to make the retrieval unit and the reading unit the same object.

The first is parent-document retrieval: index small pieces, but when one is retrieved, return the larger passage or section it came from. You match on the sentence and read the section. The second is adding context back into the chunk before embedding it — carrying headings down into each piece, or prepending a short generated sentence saying where in the document this passage sits and what it refers to. Anthropic reported this approach, which they called contextual retrieval, cut top-20 retrieval failures substantially in their tests — roughly by half with contextual embeddings and lexical search together, and further when a reranking stage was added. I would treat those specific figures as evidence from one evaluation rather than a constant, but the direction is well replicated: a chunk that names its own context is easier to find.

There is a second, quieter reason chunking dominates. Where you cut is not only about size. A fixed 500-character window is blind to the document, so it will happily slice a table in half, split a numbered list from its stem, or end a chunk mid-clause and start the next one with a dangling pronoun. Cut on the document's own boundaries instead — headings, sections, list items, table rows kept whole — and each piece is a thing rather than a fragment. Notice that this costs nothing at query time. It is entirely an ingestion decision, made once, and it changes the ceiling of every query thereafter.

Which answers the original question. No ranker can promote a passage that does not exist as a retrievable unit, or repair one that was cut in half — so chunking sets the ceiling that every later stage works beneath.

Three refinements.

First, "chunk size" is a poor summary of the decision. Two systems with an identical 800-token average behave completely differently depending on whether cuts respect structure, whether neighbouring chunks overlap, and whether headings are inherited. Size is the parameter people report; boundary placement is usually what changed the result.

Second, overlap is a patch, not a strategy. Repeating the last sentences of one chunk at the start of the next reduces the chance that an answer straddles a boundary, and it costs storage and some duplicate hits in the result list. It is worth having, but it is compensating for cuts made without regard to meaning rather than fixing them.

Third, the honest caveat: there is no established optimum, and published comparisons disagree, because the right granularity depends on the corpus. What generalises is the shape of the tradeoff and the two tricks for escaping it — not a number of tokens.

• How reranking interacts with chunk size — a slower second-pass scorer changes which compromise is affordable.
• Whether late-chunking approaches, which embed the document first and pool per passage afterwards, dissolve the tradeoff or just move it.

Every term the collection defines is gathered in the glossary.