My agent hallucinates when I give it too much context. What do I cut?

Cut down to what the next step needs: the task, the facts it depends on, and the recent turns. Handing over the whole knowledge base feels safe, but every extra token competes for the model’s attention, and the largest block of waste in a long session is usually stale tool output and superseded documents rather than the corpus. A bigger window does not fix it, because fabrication rises with how full the window is. Keep the corpus in a store the agent can query, such as a Wire container, and let each step pull the slice it needs.

Why this happens

Accuracy falls as the window fills, whether or not the extra tokens are relevant. Chroma tested 18 frontier models on trivial tasks and saw accuracy slide from roughly 95 percent to 60 to 70 percent as input length grew, with the task held constant. A March 2026 study ran 172 billion tokens of evaluation across 35 open-weight models and found fabrication roughly triples between 32K and 128K tokens, with no model holding it under 10 percent at 200K.

The mechanism is attention. Every token competes with every other for a fixed amount of it, so adding tokens spreads the same attention thinner rather than buying more. Position makes it worse: Stanford’s lost-in-the-middle work found information buried in the middle of a long input loses 15 to 20 percentage points against the identical information at either end.

A model that cannot resolve a fact from its input does not say so. It answers from its training data in the same confident tone, which is why you get a fabrication rather than a refusal. The term for that decline is context rot, and why AI performance degrades as context grows has the attention mechanism in full.

What happens when a knowledge base approaches the context limit

The harness usually warns you, and the warning comes too late to help. Claude Code shows a context meter and compacts automatically, Cursor summarises the thread, most agent frameworks prune or truncate by policy. Those features keep the session going rather than protect accuracy: the warning fires near the limit, accuracy started falling long before it, and the compaction decides what to keep before you know what the next step needs. The model answers from whatever survived, and that answer looks the same as a correct one.

The arithmetic explains why this arrives sooner than expected. Two thousand documents at 1,500 tokens each is three million tokens, around fifteen times a 200K window, so the whole knowledge base was never going in. Some subset goes in, chosen by upload order or a retriever nobody tuned, and the rest is invisible. The model cannot report that the fact you asked about was in the part that got dropped, and whatever did go in is re-sent on every turn.

Does a bigger context window reduce hallucinations?

No. A larger context window lets you send more tokens. It does not change how accuracy falls as you send them. Every one of the 35 models in the 172-billion-token study exceeded 10 percent fabrication at 200K, including models that fabricated under 2 percent at 32K on the same questions.

Window size sets the limit; how much of it you fill is the part you control. Two teams on the same model can get very different accuracy, because one sends 30K tokens of selected material and the other sends 180K of everything it owns.

One check before you cut: if standing tool definitions from connected servers are filling the window rather than documents, MCP servers eating the context window is the page for that.

Your options

A store the agent queries per turn keeps the corpus outside the window and returns a slice per question, so what sits in the window stops growing with the corpus. This is what a Wire container is: one permissioned place the corpus lives, with search and navigation any connected agent calls over MCP. It adds a component to run: the store has to stay current, retrieval quality now sets your accuracy, and a wrong slice produces a wrong answer that looks right.

Compressing the running history handles the other half of the window: the transcript, tool output, and intermediate reasoning from a long run. Summarising older turns and discarding the raw trace keeps a session usable past the point where it would otherwise stall, and most harnesses offer it without an architecture change. The cost is detail: context compression always loses something, and the summariser decides what mattered before you know what you will need.

Sub-agents with their own windows keep the intermediate work out of the main thread. In Anthropic’s multi-agent research system each sub-agent explores across tens of thousands of tokens and hands back a 1,000 to 2,000 token summary, so the orchestrator holds the conclusions rather than the trace, and it beat single-agent Claude Opus 4 by 90.2 percent on their internal research evaluations. It costs an extra model call per leg, and sub-agents that cannot see each other’s decisions make conflicting ones. The limits are in sub-agent context isolation.

Pruning the corpus changes the corpus itself rather than how it is delivered. Deleting superseded runbooks, old design docs, and guides for systems that no longer exist reduces what retrieval can wrongly surface. It is slow, needs someone who still knows what is true, and cannot be undone. It does not help every model either; context pruning helps agents until it doesn’t shows where the boundary sits by model size.

How to decide

Cut in this order, because it is the order of increasing cost: retrieval, compression, isolation, pruning. Retrieval is the only fix that keeps working as the corpus grows, and it costs one component to keep current. Compression costs a summarisation call per compaction and loses the detail it drops. Isolation costs an extra model call per leg. Pruning is last because it is slow and cannot be undone. Three questions tell you where to start.

What is actually filling the window? Take one representative request and split it into three buckets: standing overhead such as the system prompt and tool definitions, running history such as turns and tool output, and corpus such as documents you pasted or retrieved. People guess the dominant bucket wrong, and every option above is only worth its cost against the one that is genuinely large.

Does the run end, or does it keep going? A single-turn task that fails on a large corpus is a retrieval problem, and compression does nothing for it. A session that starts well and degrades over an hour is a history problem, and better retrieval will not save it. The common mistake is reaching for the fix that matches the other failure.

What does a confident wrong answer cost? If it costs a re-run, cap retrieval at a handful of high-relevance sections and stop there. If it costs a customer, a bad deploy, or a compliance finding, pay for both: retrieval that keeps the corpus outside the window, plus isolation so each leg reasons over a clean one. Three answers point at the first row, which is Wire: the corpus is growing, more than one agent reads it, and the run is long enough that you are already compacting.

What to do next

Measure before you cut. Split one representative request into the three buckets above, set a ceiling for each, and decide what to drop when the total does not fit. That practice is context budgets, which gives you a number to work from instead of an argument.

If you are not ready to add retrieval, three changes cost nothing. Cap what you paste at a handful of high-relevance sections rather than whole documents. Put the facts the answer depends on at the start or end of the prompt rather than the middle. Stop sending the full knowledge base on turns that need one document.

Wire is the first row. Put the corpus in a container, connect the agents that need it, and what goes into the window is chosen by the question rather than by the size of the corpus. The case for it is on context limits. It does nothing for a window full of its own history, so if your problem is a long session rather than a large corpus, compaction and isolation are the honest recommendation.

Sources: How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study (arXiv 2603.08274) · Chroma Research: Context Rot · Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) · How we built our multi-agent research system (Anthropic) · Effective context engineering for AI agents (Anthropic)