We are about to commit another quarter to our retrieval pipeline. Is RAG still worth it, or should we fine-tune?

Keep the retrieval and fine-tune only what retrieval cannot fix. The evidence on the fork has not moved: models absorb new facts from fine-tuning slowly, and the examples that do land push their hallucination rate up, while a retrieval index updates with a write. What changed is that fine-tuning got cheap enough to be worth doing for the narrow thing it is genuinely good at, which is the shape of the output rather than the content of it.

Is RAG outdated?

No, but the reason to keep it is narrower than it was two years ago, and three real things changed underneath the question. Context windows crossed a million tokens, so a small corpus can now sit in the prompt. Prompt caching removed most of the penalty for re-sending it, with Anthropic pricing cache reads at a tenth of the base input rate. And fine-tuning stopped being a research project: the LoRA Land technical report trained 310 adapters across 31 tasks for under 8 dollars each on average.

What did not change is the part people keep hoping did. Fine-tuning is a poor way to install facts. The canonical head-to-head ran knowledge injection both ways and found RAG scored 0.875 on a current-events set where fine-tuning managed 0.504 on the same model, with a second model showing the same split at 0.876 against 0.511. A later study explains the mechanism. Training examples carrying knowledge the model has not seen are learned much more slowly than examples matching what it already knows, and as those examples are finally learned they linearly increase the model’s tendency to hallucinate. Fine-tuning teaches a model to use what it has. It does not reliably add to it.

The bigger-window story has a similar shape. On NoLiMa, a benchmark that removes the literal word overlap between question and answer, 11 of 13 models advertising 128K support fell below half their short-context score at 32K tokens, and the strongest performer dropped from 99.3 percent to 69.7 percent. Capacity and usable capacity are different numbers. The benchmark data on long context and retrieval breaks down which tasks each one actually wins.

When does fine-tuning genuinely beat retrieval?

When the model already knows enough and only the output is wrong. That case is real, and content written by retrieval vendors tends to skip past it. The LoRA Land report measured 310 fine-tuned adapters across 10 base models and 31 tasks, and the 4-bit LoRA models beat their own base models by 34 points on average and GPT-4 by 10. Those tasks are narrow, well specified, and supplied with training data, which is exactly the profile where retrieval has nothing to contribute, because no fact is missing.

OpenAI’s own guidance points at the same set: classification, nuanced translation, generating content in a specific format, and correcting instruction-following failures, starting from roughly 50 well-crafted demonstrations. The second reason is economic rather than qualitative. A fine-tuned model needs a shorter prompt, so a high-volume task can stop carrying a few thousand tokens of instructions and examples on every call, which shows up as both cost and latency.

What that leaves out is the retrain. A fine-tune is a snapshot of your data and your preferences on the day you ran it, with no notion of which of its facts have since become wrong. It states the stale ones with the same confidence as the current ones.

What actually differs between retrieval and fine-tuning?

They are not two ways of doing one thing. Retrieval changes what the model sees at inference; fine-tuning changes how it behaves. Every row below follows from that, which is why the two combine well and substitute badly.

Your options

The retrieval pipeline you already run is the default for a reason, and the maintenance burden is usually a measurement problem rather than an architecture problem. Chunking, ranking, and query rewriting each lose recall independently, and teams without an eval set cannot tell which one is costing them. It stays the right answer whenever the corpus changes, the answer needs a citation, or different users are allowed to see different things.

Fine-tuning on your own data is cheap now and good at a specific job: making the model respond in a fixed shape, in your register, or on a narrow task you can describe with examples. It fails when it is used as a knowledge store, because facts learned this way arrive slowly, cannot be updated without another training run, and raise hallucination rates on the way in.

Retrieval plus a fine-tuned model is where most mature systems land, with retrieval handling the facts and training handling behavior. Berkeley’s RAFT recipe goes further and trains the model on retrieval-shaped inputs, teaching it to ignore distractor documents and quote from the useful ones. The cost is two subsystems to evaluate, and it is premature until retrieval is already accurate.

Neither, and prompt the base model as it is deserves an honest test before either project starts. Frontier models carry a great deal of public domain knowledge, and a measurable share of what teams build retrieval for is knowledge the model already has. Run 50 real questions cold. If it answers them, the pipeline was never the missing piece. It stops working the moment the answer depends on your data or on anything after the training cutoff.

A managed container your agents search keeps retrieval’s properties without the pipeline underneath it: one permissioned place agents read and write, with chunking, embeddings, and per-user scoping handled for you. Wire is one implementation. It is the wrong choice if you need a particular vector database, a custom ranker, or retrieval that never leaves your own network.

How to decide

Is the failure a missing fact or a wrong shape? Label fifty failed answers by hand and split them. If the right passage was never retrieved, no amount of training fixes it. If the passage was in context and the answer still came out wrong, that is a behavior problem and fine-tuning is the cheaper instrument. Most teams have never run this split, which is why retrieval failures get diagnosed as reasoning failures.

How often does the answer change, and who is allowed to see it? Anything that changes more than quarterly makes fine-tuning a treadmill. Anything scoped per customer, per tenant, or per role rules it out on its own, because weights have no access boundary and no way to cite what they drew on.

What happens when you swap base models? Retrieval survives it. Weights do not. If you expect to move to a better model within the year, an investment in retrieval carries forward and an investment in training does not, which is worth pricing before either decision.

What to do next

Build the eval set before the architecture. Fifty real questions with known good answers, run first against the bare model and then against your current pipeline, tells you which of the options above is even addressing your problem. If that set shows retrieval is finding the wrong material rather than presenting it badly, the fix is upstream of both choices here, and what the research says about chunk size is the cheapest place to start.

If the set instead confirms retrieval is the right instrument and the pipeline is the part you would rather not own, a managed container is worth pricing against another quarter of in-house work. Wire runs the chunking, embeddings, and per-user scoping behind one permissioned place every agent reads and writes, and how its retrieval measures up is published rather than asserted. It is not the answer if the pipeline has to live in your own network or on a vector database you have already standardized on.

Sources: Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (arXiv 2312.05934) · Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (arXiv 2405.05904) · LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4 (arXiv 2405.00732) · NoLiMa: Long-Context Evaluation Beyond Literal Matching (arXiv 2502.05167) · RAFT: Adapting Language Model to Domain Specific RAG (arXiv 2403.10131) · OpenAI supervised fine-tuning guide · Anthropic prompt caching documentation