Chunking Strategies Decide What Context AI Sees

Most teams treat chunking as a preprocessing chore: pick a token count, add a little overlap, move on. But the way you split a document is the single decision that controls which slices of it ever reach the model. An AI system never sees your document. It sees the chunks your retriever pulled back, and nothing else. If a boundary cuts a definition off from the clause that qualifies it, the model answers from half an idea.

That is why chunking strategies are a context engineering decision, not a data-prep afterthought. The chunk is the unit of context delivery. Choose it badly and even a perfect embedding model and a perfect prompt cannot recover the missing half of the thought. Choose it well and you hand the model complete, self-contained units it can actually reason over. This post walks through what chunking does to context, how the main strategies compare on recent benchmarks, and how to pick one for your own corpus.

What chunking actually does to context

Chunking sets the smallest unit of meaning your retriever can return, which means it sets the floor on context quality. A retriever ranks chunks and returns the top matches; the model only ever sees those passages. So the chunk boundary is a hard constraint: any information that gets separated from its chunk is invisible at answer time unless a second chunk happens to carry it.

This is easy to underestimate because the failure is silent. A fixed 300-token cut that lands in the middle of a sentence does not throw an error. It produces a chunk that embeds slightly off-topic, ranks slightly lower, and returns a passage that is technically related but missing the qualifier that changes the answer. The model does not know a clause is missing. It generates a confident, fluent response built on a fragment, which is one of the most common ways retrieval quietly fails. Good chunking is the work of making each unit small enough to retrieve precisely and complete enough to stand on its own.

Chunking strategies at a glance

There is no single best chunking strategy, but the main approaches trade off the same two things: how cleanly they preserve meaning, and how much they cost to compute. The table below summarizes where each fits.

Most production systems start with recursive splitting at 256 to 512 tokens with light overlap, then move to semantic, document-based, or late chunking only when evaluation shows the simpler method is losing answers.

Why semantic chunking isn’t automatically better

Semantic chunking sounds like it should win, but it does not reliably beat simpler methods. The intuition is appealing: split where the topic changes, not at an arbitrary token count. In practice, a study titled “Is Semantic Chunking Worth the Computational Cost?” found that semantic chunking delivered no consistent accuracy advantage over fixed-size splitting while costing far more to compute. A 2026 benchmark across seven strategies and 50 academic papers reinforced the point: recursive 512-token splitting led at 69% accuracy, while semantic chunking trailed at 54% and produced fragments averaging just 43 tokens.

The cost gap is real too. Benchmarks put semantic chunking at roughly 14 times slower than token-based splitting, because it has to embed every sentence to find the boundaries before it can embed the chunks. That is a steep price for a method that often matches, rather than beats, a cheap baseline. Semantic chunking earns its cost on long, sprawling documents where topic boundaries sit far apart and a fixed cut would routinely merge unrelated sections. On dense, well-structured material, recursive or document-based splitting usually gets you there for a fraction of the compute.

The boundary problem is a retrieval-failure problem

Bad chunk boundaries surface as retrieval failures, and retrieval failures are measurable. Anthropic’s work on contextual retrieval put a number on it: a standard chunking and embedding pipeline failed to return the right passage in its top 20 results 5.7% of the time. Those misses are not random. They cluster on chunks that lost the context that made them findable, like a paragraph that says “the limit was raised to 10,000” without the chunk ever naming which limit or which product.

When the right passage does not come back, the model does not abstain. It produces an answer from whatever did come back, which is how a chunking flaw turns into a hallucination downstream. The longer the surrounding context the model has to stitch together from fragmented chunks, the worse this gets, because attention degrades across long, low-signal inputs in the pattern known as context rot. Fixing the boundary at ingestion is cheaper than trying to repair a fragmented answer at generation time.

Putting the context back: contextual and late chunking

The strategies that move the needle most are the ones that put back the context a boundary strips out. Anthropic’s contextual retrieval prepends a short, chunk-specific description to each chunk before embedding it, so a passage about a raised limit carries the document and section it came from. That single change cut the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%. Adding a contextual keyword index brought the reduction to 49%, and layering a reranker on top reached 67%, down to a 1.9% failure rate. Every one of those gains came from restoring context that naive chunking discarded.

Late chunking attacks the same problem from the embedding side. Instead of splitting text and embedding each piece in isolation, it runs the whole document through a long-context embedding model first, then splits the resulting token embeddings into chunks. Because every token was embedded while the model could see the entire document, each chunk embedding carries information from its neighbors, and the method stays robust even when the boundaries themselves are imprecise. The lesson across both techniques is consistent with everything we know about structured context: a retrieved unit works best when it arrives self-contained, and chunking that preserves surrounding context also makes semantic search more precise.

How to choose a chunking strategy

Pick your chunking strategy by testing candidates against your own queries, not by copying a default from a tutorial. The right answer is corpus-dependent, and the gaps between strategies are large enough to matter. In a clinical decision-support study, chunking aligned to logical topic boundaries roughly doubled retrieval F1 over a fixed-size baseline (0.64 versus 0.24) on the same questions and the same model, changing only how the documents were split. That is the size of swing chunking can produce, and you will not know which method wins on your data until you measure it.

A practical path:

1. Start with recursive splitting at 256 to 512 tokens with 10 to 20 percent overlap. It is cheap and competitive.
2. Respect document structure when you have it. Split on headings, tables, and code blocks before falling back to token counts, so logical units stay intact.
3. Chunk at ingestion, not per query. Splitting once, with the whole document in view, keeps structure intact and avoids repeating the work on every request. Systems like Wire process files into structured units at upload time, so an agent retrieves passages that were split while the full document and its relationships were still visible, instead of text re-cut blindly at query time.
4. Reach for semantic or late chunking only when evaluation shows simpler methods are fragmenting answers, and you can afford the extra compute.
5. Measure on real queries. Track retrieval accuracy and hallucination rate, not chunk aesthetics. The boundary that reads cleanly to a human is not always the one that retrieves best.

Chunking as context engineering

Chunking is where context engineering starts, because it decides the raw material every later stage works with. Embeddings, reranking, prompt formatting, and the size of your context window all operate on chunks they did not choose. If those chunks are fragments, no amount of downstream tuning recovers the missing half of the thought, which is part of why the long-context-versus-RAG debate keeps circling back to retrieval quality rather than raw window size.

The teams getting reliable answers out of retrieval are not the ones with the cleverest chunking algorithm. They are the ones who treat the chunk as a unit of meaning to be preserved, measure what their boundaries do to retrieval, and put back the context that splitting strips out. Chunk for the model that has to reason over the result, not for the splitter that produces it.

Sources: Anthropic: Contextual Retrieval · Jina AI: Late Chunking in Long-Context Embedding Models · Late Chunking (arXiv 2409.04701) · Is Semantic Chunking Worth the Computational Cost? (arXiv 2410.13070) · A Systematic Investigation of Document Chunking Strategies (arXiv 2603.06976) · Comparative Evaluation of Advanced Chunking for Clinical Decision Support (MDPI Bioengineering) · Firecrawl: Best Chunking Strategies for RAG in 2026