RAG vs fine-tuning: when to use each

When teams ask whether to use RAG or fine-tuning, they usually frame it as a choice between two tools. The research suggests a different framing. RAG and fine-tuning solve different problems, and the people who get the most out of either one have already made the harder decision: what does the model actually need from the system around it?

One measurement captures the gap. A January 2026 study comparing the two approaches on multi-hop question answering with novel knowledge, run across three 7B open-source models, found RAG more than doubled accuracy over supervised fine-tuning on the same tasks. The answer depends on what you are trying to change.

What each approach actually does

RAG and fine-tuning change different parts of the system. RAG leaves model weights alone and injects relevant documents into the prompt at inference time. The model reads the retrieved context and generates an answer from it. Fine-tuning does the opposite: it adjusts the model weights using a domain-specific dataset so the model’s default behavior shifts toward that domain.

This distinction matters because it defines what each approach can change. Fine-tuning is good at changing behavior, style, tone, format, and narrow task specialization. RAG is good at giving the model access to information it was never trained on, or information that changes faster than any realistic retraining cadence. A model fine-tuned on 2024 data does not know what happened in 2026 unless you fine-tune again. A RAG system with a fresh index does.

Weights versus prompts is also why cost profiles differ. Fine-tuning pays upfront in training compute and ongoing in retraining cycles. RAG pays per query in retrieval and in extra tokens injected into the context. Each has a regime where it dominates, and those regimes do not overlap as much as the debates suggest.

Comparison at a glance

Read the table as a shorthand, not a verdict. Most production systems mix approaches, and the question is which one carries the weight for a given behavior.

When RAG wins

RAG wins when the knowledge needs to change, needs to be attributed, or needs to be scoped. The 2026 multi-hop QA comparison tested unsupervised fine-tuning, supervised fine-tuning, and RAG across three 7B models and found RAG more than doubled fine-tuning’s accuracy on questions requiring knowledge not seen during training. A separate 2025 medical LLM study on the MedQuAD dataset reached the same conclusion across five model families (Llama-3.1-8B, Gemma-2-9B, Mistral-7B, Qwen2.5-7B, Phi-3.5-Mini): RAG and hybrid RAG-plus-fine-tuning consistently beat fine-tuning alone, with the gap widest on questions requiring fresh factual knowledge.

Freshness is the most common reason to choose RAG. A support bot that references product documentation cannot afford to wait for quarterly retraining each time the docs change. A sales agent that summarizes deal activity needs today’s emails and calls, not last week’s snapshot. Anything downstream of a changing source of truth pays a real tax under fine-tuning: the model is always slightly out of date, and the tax compounds as the domain drifts.

Provenance is the second reason. Regulated industries, customer-facing applications, and any use case where a wrong answer has a cost need to be able to show where an answer came from. RAG returns the source documents alongside the generation. Fine-tuning offers no equivalent trail, and trying to reconstruct one after the fact tends to produce confident-sounding but unreliable attributions.

Scope is the third. Multi-tenant applications need each user to see only their own data. Per-user fine-tuning is impractical at any meaningful scale. Retrieval lets you filter the index by tenant, role, or access grant so the model only ever sees authorized context. Structured context with explicit metadata makes this filtering precise rather than best-effort.

When fine-tuning wins

Fine-tuning wins when the behavior needs to change, not the facts. If the model’s default output is close but consistently wrong on format, tone, or task structure, fine-tuning reshapes those defaults in a way that prompting and retrieval cannot. A model fine-tuned on a few thousand examples of your preferred JSON schema will emit that schema reliably without long system prompts or repeated examples in every call.

Narrow, high-volume tasks are the clearest case. If you run 100,000 queries per day on a well-defined task, such as classifying support tickets, extracting structured fields from contracts, or routing intents, a fine-tuned smaller model can cut inference cost and latency dramatically compared with a large general-purpose model plus long retrieved context. The training cost amortizes quickly at that volume.

Style and voice are the other strong case. Brand writing guidelines, legal document conventions, and internal code style are easier to teach through examples than through prompt instructions. Anthropic and OpenAI both recommend fine-tuning primarily for style, format, and behavior, and both recommend RAG first when the task is about knowledge rather than behavior.

A real cost comparison from a 2026 production write-up shows the dynamic. For a B2B SaaS workload running 2,000 queries per day across 500 pages of documentation, RAG costs about $18,400 in year one, while fine-tuning costs about $30,600 after setup, hosting, and quarterly retraining. At 100,000 queries per day on a narrow classification task, those numbers invert. The decision is volume-dependent, not ideology-dependent.

When the right answer is both

The hybrid pattern is winning in production because each approach covers a different weakness. Berkeley’s RAFT recipe (Retrieval Augmented Fine-Tuning) trains a model on retrieval-style inputs: given a question plus a set of documents, some relevant and some distractors, the model learns to ignore the irrelevant ones and cite directly from the useful ones. A more recent ICLR 2026 study extends this by using RAG-based distillation to internalize new skills: student models reached roughly 91 percent success on ALFWorld versus 79 percent for baselines, while using 10 to 60 percent fewer tokens than retrieval-augmented teachers at inference. Hybrid approaches keep the freshness advantage of retrieval while shaping model behavior through training.

The canonical hybrid split looks like this. Fine-tune the model on style, output format, tool-use patterns, and domain-specific behaviors that are stable. Keep knowledge out of the weights. Then use RAG at inference for anything that changes, needs provenance, or is scoped per user. The model handles behavior; the retrieval system handles facts.

This pattern also reduces the common failure mode of each approach. A pure RAG system with a general model often has the right documents but the wrong output format, because the model defaults fight the task. A pure fine-tuning system often has great format but stale or missing facts. Splitting the responsibilities gives each subsystem a narrower job and makes the failures easier to diagnose.

The context engineering lens

Neither RAG nor fine-tuning solves the core problems of context engineering. Both sit on top of decisions about what the model should see, when, from where, and with what level of trust. Skip those decisions and both approaches fail in predictable ways.

Context rot degrades accuracy as input length increases. RAG systems that dump 50 retrieved chunks into a prompt trigger it every bit as reliably as full-document long-context approaches. Fine-tuning does not help. The fix is to retrieve less, not retrieve more, and to structure what you do retrieve so the important signal sits where attention is strongest.

Context poisoning is a worse problem for fine-tuning than for RAG. Bad data in a training set becomes ground truth baked into weights. Bad data in a retrieval index can be corrected by removing the source or adjusting the filter. Provenance is a debuggability advantage that shows up late in a project, often during the first incident, and it tends to decide whether teams trust their own system.

Scope leakage is another shared failure. If the retrieval system returns documents the user should not see, or if the fine-tuned model was trained on cross-tenant data, the same access problem shows up in different clothing. This is why AI systems that handle multiple customers benefit from scoping context at the source. Platforms like Wire take this approach by isolating each container with its own access controls, so an agent querying a container retrieves only entries it is authorized to see, regardless of whether those entries later feed a prompt or a training set.

Freshness is the dimension that usually decides. If the underlying information changes faster than your retraining cadence, fine-tuning cannot keep up no matter how good the training data is. Retrieval plus caching handles this well at most scales, and the few scales where fine-tuning genuinely wins on cost are narrow enough that the decision usually makes itself.

A decision checklist

Walk through these six questions before picking either approach.

1. Does the underlying information change more than once a quarter? If yes, lean RAG.
2. Do you need source citations for any answer? If yes, lean RAG.
3. Is the content scoped per user, tenant, or role? If yes, lean RAG.
4. Is the model close but consistently wrong on format, style, or narrow task structure? If yes, lean fine-tuning.
5. Are you running more than 100,000 queries per day on a narrow, stable task? If yes, consider fine-tuning for cost.
6. Do you need both fresh knowledge and reliable formatting? Combine them, with fine-tuning for behavior and RAG for facts.

The framing worth keeping is the one the research points at. RAG and fine-tuning are not competitors for the same job. They are tools for different jobs that sit on top of a context architecture. Decide the architecture first, and the choice between them usually becomes obvious.

Sources: Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge (arXiv 2601.07054) · Fine-tuning with RAG for Improving LLM Learning of New Skills (ICLR 2026, arXiv 2510.01375) · Medical LLMs: Fine-Tuning vs. Retrieval-Augmented Generation (MDPI 2025) · RAFT: Adapting Language Model to Domain Specific RAG (arXiv 2403.10131) · RAG vs Fine-Tuning: The Real Cost Comparison for 2026 · Should I use Prompting, RAG or Fine-tuning? (Vellum)