What is Chunking?

How chunking works

Chunking splits a document into passages before anything is embedded or indexed. A splitter walks the text and cuts it according to some rule: a fixed token count, a hierarchy of separators like paragraph then sentence then word, the document’s own structure such as headings and tables, or a signal derived from the content itself like a drop in embedding similarity between adjacent sentences. Each resulting chunk is embedded and stored, and retrieval then ranks chunks rather than documents.

The consequence is that the chunk, not the document, is the unit of retrieval. A model asking a question never sees your file. It sees whichever passages ranked highest, exactly as they were cut.

Why chunking matters

Chunking sets the floor on context quality, and it does so silently. A boundary that lands mid-argument does not raise an error. It produces a chunk that embeds slightly off-topic, ranks slightly lower, and returns text that is topically related but missing the qualifier that changes the answer. The model has no way to detect the omission, so it answers fluently from a fragment.

Anthropic’s contextual retrieval work put a number on the base rate: a standard chunking and embedding pipeline failed to return the right passage in its top 20 results 5.7% of the time. Prepending a short chunk-specific description before embedding cut that to 3.7%, a 35% reduction, and adding contextual BM25 with a reranker brought it to 1.9%, a 67% reduction. Every one of those gains came from restoring context that the split had discarded, which is the clearest signal available that boundary damage is real and measurable. Chunking strategies and what they do to context walks through contextual and late chunking in full.

Because chunking happens at ingestion, the decision is durable. A poor split is baked into every query the index serves until you re-chunk and re-embed the corpus.

What the research actually says

The literature does not converge on a single best strategy, and the disagreements are informative rather than noise.

Is Semantic Chunking Worth the Computational Cost? (Qu, Tu and Bao, NAACL 2025 Findings) evaluated semantic chunking across document retrieval, evidence retrieval and retrieval-based answer generation. It found the gains inconsistent and context-dependent, appearing mainly on synthetic datasets stitched together for high topic diversity, while fixed-size chunking often performed better on documents that resemble real ones.

Chunk Twice, Embed Once (Amiri and Bocklitz, 2025) swept 25 chunking configurations across five method families for chemistry retrieval and found recursive token-based chunking at 100 tokens with no overlap outperformed the alternatives, with minimal resource overhead.

A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity (Shaukat, Adnan and Kuhn, 2026) went the other way. Across 36 segmentation approaches, six knowledge domains and five embedding models, paragraph group chunking reached mean nDCG@5 around 0.459 with Precision@1 near 24%, while a fixed-size character baseline sat below 0.244 nDCG@5 and 2 to 3% Precision@1.

Those results are reconcilable but only if you read them carefully. Fixed-size splitting on tokens, respecting separators, is a strong baseline. Fixed-size splitting on characters, ignoring structure, is not. And the winning configuration is corpus-dependent enough that a result from chemistry papers does not transfer to support tickets. What chunk size you should use for RAG works through all three studies, reconciles them, and links the papers directly.

Common misconceptions about chunking

• “Semantic chunking is the sophisticated choice.” It is the expensive choice. It has to embed every sentence to find boundaries before embedding the chunks, and the peer-reviewed result is that the cost is often not repaid.
• “Overlap is free insurance.” Overlap duplicates content across chunks, inflating index size and surfacing near-identical passages in the top-k. The best configuration in one 25-way sweep used none.
• “Bigger chunks are safer because they hold more context.” Larger chunks dilute the embedding, so a passage matches more queries less precisely, and they consume more of the window per result.
• “You can fix chunking at query time.” The index only contains what the splitter produced. Query-time reranking can reorder bad chunks but cannot restore information a boundary removed.

Chunking and Wire

Wire chunks at ingestion, while the whole document and its relationships are still in view, rather than re-cutting text per request. Files are processed into structured entries with relationships and embeddings at write time, so an agent calling wire_search retrieves passages that were split with full-document context available, and can then call wire_navigate to walk from a match into related entries instead of hoping a single chunk carried everything.

That ordering matters for the failure mode above. The information most often destroyed by a boundary is the link between a passage and the thing it refers to, and a retrieved entry that keeps its relationships does not depend on that link surviving inside the chunk text.