Demand paging for the AI context window

The context window is usually treated as the place where an agent’s information lives. A 2026 systems paper from the University of British Columbia argues that framing is the root cause of runaway agent costs. In The Missing Memory Hierarchy: Demand Paging for LLM Context Windows, Tony Mason measures real agent traffic and concludes that “the context window of a large language model is not memory. It is L1 cache: a small, fast, expensive resource.” Across 4.45 billion tokens of production traffic, 21.8% of what sat in those windows was structural waste, content that entered once and was recomputed on every turn without ever being referenced again. The fix is a 70-year-old operating systems idea: demand paging. This post explains what that means for context window management, why the economics are stranger than they look, and how to design a paging policy that keeps an agent’s working context small and current.

The context window is your smallest and most expensive memory tier

The context window behaves like L1 cache, not like main memory, and pricing it that way changes every decision downstream. L1 cache is fast and close to the processor but tiny and costly, so good systems keep only the active working set there and push everything else down the hierarchy. An LLM context window has the same shape: a 200K-token window is small relative to an agent’s full state, and attention cost grows quadratically with how full it is, so a window packed with rarely-referenced tokens is the most expensive way to store anything.

The waste is measurable. The UBC study classified 21.8% of tokens across 4.45 billion as structural waste, broken down into dead tool output at 26.5% of that waste, unused tool definition schemas at 20.2%, and static system content at 11.0%. It also reports a median amplification factor of 84.4 times for main agent sessions, meaning the same stale content was reprocessed dozens of times after it left the working set. This is the cost of treating the window as a place to accumulate context rather than a tier to manage, and it is the same accumulation problem behind why long-running agents forget mid-task and lose the plot.

What demand paging means for an agent

Demand paging means an agent keeps only its working set resident in the window and loads other content from cheaper storage the moment it is referenced again. In an operating system, a program’s full address space lives on disk, and only the pages it actively touches are loaded into RAM; touching a page that is not resident triggers a page fault that loads it. Mapped onto an agent, the window is RAM, persistent state is the backing store, and a reference to evicted content is a fault that pages it back in. The mapping is direct enough to use as a design vocabulary.

The study implements this as a transparent proxy called Pichay that sits between the client and the model API, the way a memory management unit sits between a program and physical memory. It organizes context into four tiers: the active generation window, a demand-paged working set, compressed session history, and cross-session persistent memory that is indexed and retrievable. This is the structural version of context offloading: rather than shrinking text in place, you move it to a lower tier and pull it back through a handle when it is needed. Offline, paging proved nearly free, with a fault rate of 0.0254% across 1.4 million simulated evictions, because real agent sessions have strong locality.

The economics are inverted, so evict aggressively

The single most important difference from hardware is that keeping context resident is not free, so the correct default is to evict early. In a CPU, a page sitting in RAM costs nothing until you touch it, which is why operating systems keep pages around speculatively. In an LLM, every token kept in the window is fed back through attention on every subsequent turn, so a resident token costs money and compute repeatedly whether or not it is used. The study frames the optimal rule plainly: evict any content that will not be referenced for more than about one turn, and fault it back if you were wrong.

The agent-level evidence backs this up. In Less Context, Better Agents, a 50-task enterprise benchmark compared full-history retention against a recency window plus a compact running summary.

Keeping everything was the worst configuration on every axis. Aggressive eviction with a small summary raised completion from 71.0% to 91.6% while cutting token use by roughly 63%, and it dropped stale-state reference errors from 34 to 4 across the pooled runs. The pattern held on a second model, where Claude Sonnet 4.5 went from 88.0% on full context to 94.5% with pruning plus summarization. Eviction is not a cost-saving compromise that trades away quality. Past a point, accumulation is what degrades quality, which is the through-line of our piece on context budgets and how to allocate tokens.

Why naive approaches thrash

The two common responses to a full window, dump everything or re-retrieve everything, both fail, and the second fails loudly. Dumping everything into a large window is the no-hierarchy approach, and it produces exactly the amplification the UBC study measured, since stale tokens get recomputed every turn at quadratic cost. It also runs into the accuracy ceiling documented in long context versus RAG, where burying relevant content in a large window lowers retrieval accuracy regardless of how much you pad around it.

Re-retrieving everything every turn is worse, because it reintroduces thrashing, the failure mode where the evict-and-fetch cycle consumes more work than it produces. The study observed it directly: one 681-turn session was brought from 5,038KB down to 339KB of context, but at one point hit a 97% fault rate, the agent equivalent of a machine that spends all its time paging and none of it computing. Pure compression has a different limit. Methods that shrink text are lossy and model-dependent, and the paper notes that cached tokens, the supposed escape hatch, “still occupy the context window and require attention computation for every output token generated.” Prompt caching cuts the cost of re-sending tokens, which is real and worth using, as we cover in how prompt caching cuts agent costs, but a cached token is still a resident token competing for attention.

Designing the paging policy

A good paging policy combines aggressive eviction with cheap recovery, so the agent can drop content freely without losing access to it. Four design choices from the research do most of the work.

Use retrieval handles as anchors. When content is evicted, leave a stub in its place, something like [Paged out: Read file.py (8,192 bytes). Re-read if needed.]. The study found models recognize these unprompted and re-request content on their own, which turns eviction into a late-binding pointer rather than a deletion. This is the same instinct behind treating MCP resources as addressable handles rather than inlined blobs, an idea we expand in MCP resources are half of MCP.

Pin by reference, not by recency. Plain LRU fails for agents because timestamps go stale while the underlying file stays relevant; a file read during planning often matters for the rest of the session. Fault-driven pinning, where one fault keeps a page resident for the working set, implements Denning’s 1968 working-set model and matches how agents actually use context.

Let the model cooperate. Unlike a CPU program that cannot see its own memory pressure, an agent can tell you what it no longer needs and act on it, which the paper enables through release signals like phantom tools and inline cleanup tags, the cooperative version of the memory-as-tools pattern. That loop falls out naturally when the backing store is a context container the agent can write to: connected to a Wire container, an agent can summarize a finished sub-task and write that summary plus a pointer back with wire_write, a free call that returns a durable entry handle, then drop the raw turns from its window and page the material back later with wire_search, which treats prior agent writes as first-class results, or by resolving the handle directly. The agent runs the paper’s evict-with-anchors policy itself with standard tools, rather than depending on the transparent proxy the authors had to build to do it on the model’s behalf (how Wire handles context limits).

Get more conservative under pressure, not less. Because attention is quadratic, a fault is cheap when the window is near-empty and expensive when it is near-full. The counterintuitive guidance is that eviction should slow down as the window fills, since paging at 180K tokens costs far more than paging at 20K. Anthropic’s context engineering guide reaches a compatible conclusion from the prompt side: the goal is the smallest set of high-signal tokens that still produces the outcome, applied continuously rather than once.

Takeaways

Treat the context window as the top of a hierarchy, not as storage. Hold the working set in the window, push everything else to a backing store, and page content in on demand through stable handles. Because resident tokens are recomputed every turn, evict early by default and rely on cheap faults to recover anything you dropped too soon. Watch the fault rate the way you would watch a thrashing machine: a high evict-and-refetch ratio means your working set is mis-sized, not that paging is wrong. The practitioners getting long-horizon agents to work are not buying bigger windows. They are doing context engineering on the window they have, managing it like the scarce, expensive cache tier it actually is.

Sources: The Missing Memory Hierarchy: Demand Paging for LLM Context Windows · Less Context, Better Agents · Anthropic: Effective context engineering for AI agents