Use RAG when answers depend on sources that change or that only some users may see. Consider fine-tuning when a measured quality gap remains after you've improved the prompt and the workflow, usually in how the model does the task, such as output format, task steps or tool use. Use both when the model needs current facts and better task behavior. Test retrieval and generation separately before you pay for either. The table below covers 10 common situations, and the last sections cover the costs and the eval to run first.
What RAG does
Retrieval-augmented generation looks up documents when a request arrives and adds them to the prompt. The method comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. A basic pipeline splits documents into chunks, embeds them and stores the vectors in an index, then embeds each request and pulls the closest chunks. Production systems often add keyword or hybrid search, metadata filters and a reranker before the context is assembled.
RAG suits facts that change. Updating knowledge means re-indexing a document, which is far quicker than a training run, and deleting a document removes it from future answers. The retrieved passage gives the model a source to cite, though you still need to check that the answer matches it. With permissions enforced at retrieval, each user only gets documents they're allowed to see. The weak point is retrieval itself: if the right chunk doesn't come back, the model answers without it. Every retrieved token is also billed as input on every request. Retrieved instructions or tool docs can help with behavior too, but retrieval doesn't reliably change how a model does a task it keeps getting wrong.
What fine-tuning does
Fine-tuning trains an existing model on examples of the task. Supervised fine-tuning (SFT) shows the model correct outputs to copy. Reinforcement learning, such as GRPO, lets the model try the task and rewards the attempts that score well, which needs a reward you can compute. Many teams train a small adapter with LoRA and leave the base weights frozen, which keeps the GPU bill down.
Fine-tuning suits behavior: a fixed output format, the steps of a multi-turn task, when to call a tool and when to stop, or a house style. A fine-tuned model may also need shorter prompts, since some instructions and examples move into the weights. And a small model trained on one narrow task can match or beat a much larger general model on that task at a fraction of the cost per request.
Weights are usually the wrong place to store facts that change or that you need to attribute. In Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, a 2024 study from Google and the Technion, models in the authors' question-answering experiments learned training examples with new facts more slowly than examples consistent with what they already knew, and as they learned the new facts they became more likely to hallucinate. Domain adaptation and continued pretraining are different setups, but any fact trained into weights stays as it was until the next training run.
The decision table
| Situation | Start with | Why |
|---|---|---|
| Answers depend on documents that change every week, such as policies, prices or tickets | RAG | Re-indexing is quicker than retraining, and you can see and delete what the model draws on. |
| Users need to see the source behind each answer | RAG | The retrieved passage is a source to cite. Check that answers match it. |
| Different users may see different documents | RAG | Retrieval can filter by permission. Shared weights can't keep a fact from one user. |
| Output must follow a strict format every time | Structured outputs, then fine-tuning | Constrained decoding handles the syntax. If the content inside the format is still wrong, training can help. |
| A multi-step task where the model has the facts and takes the wrong steps | Fine-tuning, with RL if you can score the outcome | Training on full trajectories targets the sequence. In our claims study the trained model used 44% fewer searches. |
| A narrow, high-volume task where the frontier model's bill is the problem | Fine-tune a small model | In the claims study a trained 4B model cost $1.75 per 1,000 claims against $17.86 for Claude Haiku 4.5. |
| A throughput or speed budget the current model misses | Fine-tune a small model | The same 4B model processed 1,000 claims in 0.35 hours against 3.30 hours for Haiku 4.5. That's total processing time, so measure per-request latency at your own load. |
| A house style or domain vocabulary the model keeps missing | Fine-tuning | Tone and word choice are behavior, and examples in the prompt cost tokens on every call. |
| A handful of examples and no eval yet | Prompting, for now | Neither method can be judged without an eval. Build that first. |
| Facts that change and a procedure the model gets wrong | Both | Retrieval supplies the facts and fine-tuning teaches the model to use them. |
The last row has its own research line. RAFT, from UC Berkeley in 2024, fine-tunes a model on questions paired with retrieved documents, some relevant and some not, so it learns to answer from the right passage and ignore the rest.
What each one costs
RAG's build cost is the pipeline: chunking, embeddings, an index, and an eval for retrieval quality. Its running cost adds to every request, because the retrieved context is billed as input. At an example price of $0.30 per million input tokens, adding 4,000 tokens of context to 1 million requests is 4 billion tokens, or $1,200, before the embedding calls and the index.
Fine-tuning's build cost is front-loaded: collecting and cleaning examples, building the eval, and GPU time for several training runs, since settings get tuned between runs. A fine-tuned model still costs money per request, so the saving comes from what you serve. If the trained model is smaller than the one it replaces, each request can get cheaper, as in the claims study's $1.75 against $17.86 per 1,000 claims. The ongoing cost is retraining when the task drifts or a better base model ships, plus RAG on top if the task also needs current facts.
Once the fine-tuned model meets your quality bar, the break-even is the up-front cost divided by the saving per request. For example, $10,000 of data work and training that saves $0.01 a request pays back after 1 million requests. If quality isn't there yet, or the saving is 0, there's no break-even to compute.
How to tell prompting has hit its ceiling
Try prompting first, since it costs the least to change. These are the signs you've reached its limit:
- Your eval score has stopped moving across several prompt rewrites.
- Few-shot examples help, and the prompt is now long enough that every request pays for them.
- The failures repeat in kind: the same format slip, the same wrong tool, the same early stop.
Read the failures before you pick a fix. Failures from facts the model didn't have, or had out of date, point to retrieval. Failures in what the model did with information it had point to the prompt, the workflow, a stronger model or training.
The eval to run before you decide
- Pull real requests from your logs, enough that a difference you care about would show. The claims study scored 500 held-out claims.
- Split them into a development set you iterate on and a test set you only score at the end. Keep related requests, such as the same customer or the same document, on the same side.
- Run the current model on the development set and label each failure: missing or stale knowledge, wrong behavior with the right information, or a task beyond the model.
- Rerun the failures with the correct document pasted into the prompt by hand. If the model now succeeds, look at retrieval and how the context is assembled. If it still fails with the answer in front of it, look at generation: the prompt, the workflow, the model's capability, and then training.
- Score cost and latency per successful task, so a cheaper model that fails more often doesn't look better than it is.
- Keep both sets out of any training data, and use the test set once, to judge the final model.
Step 4 tells you where the failure sits. It narrows the choice, and the development set is where you test each fix.
How fine-tuning works on Experiential
You can fine-tune a model with Experiential today. Book a call from the fine-tuning page with one workload in mind, and turn on prompt and response logging so the gateway captures your agent's traces. We tell you when there's enough data for a training run, and you start the run. Our team uses your traces to simulate your production environment around the tasks that matter most to your agent, then fine-tunes an open small language model with reinforcement learning in that simulation. The trained model goes back on the gateway under a new model name, and you call it through the same API. If the model ships, the training fee goes toward its inference.
The claims research study is a public example, run on a benchmark. We trained Qwen3.5-4B with GRPO and a LoRA adapter on complete search-and-reasoning trajectories. On 500 held-out development claims from AVeriTeC, its verdict accuracy was 10.9% higher than Claude Haiku 4.5's in relative terms, with a paired confidence interval of +4.5% to +17.3%. On a second set of 489 held-out claims, the trained 4B model alone scored 71.1% at $1.75 per 1,000 claims, up from 57% before training, and always using Claude Opus 5 scored 77.5% at $89. A router trained on 2,896 separate claims sent 76% of claims to the 4B model and 24% to Opus, which scored 73.6% at $27 per 1,000 claims.
To start a run, book a call from the fine-tuning page and bring the workload you want a model for.
Questions answered
Is RAG better than fine-tuning?
Neither is better in general. RAG suits knowledge that changes or needs permissions, and fine-tuning suits behavior such as format, task steps and tool use. To see where a failure sits, paste the right document into the prompt by hand: success points to retrieval, and failure points to generation.
Can you use RAG and fine-tuning together?
Yes, and many production systems do. Retrieval supplies current facts at request time, and fine-tuning teaches the model the task procedure and how to use retrieved passages. The RAFT method from UC Berkeley trains models on questions paired with relevant and irrelevant documents for this reason.
Does fine-tuning add new knowledge to a model?
It can, but weights are usually the wrong place for facts that change or need a source. In a 2024 Google and Technion study, models learned new facts from fine-tuning slowly, and learning them made the models more likely to hallucinate.
When should you fine-tune instead of improving the prompt?
When prompt rewrites stop moving your eval score, when few-shot examples make every request expensive, or when the same kind of failure keeps repeating with the right information in the prompt.
Can Experiential fine-tune a model for me?
Yes. Book a call from the fine-tuning page with one workload. Experiential's team trains an open small model with reinforcement learning on a simulation built from your gateway traces, then deploys it on the gateway under a new model name.
Is fine-tuning cheaper than RAG?
It costs more up front and can cost less per request. RAG adds input tokens to every call. A fine-tuned small model that replaces a large one can cut the price of each request, and the break-even is the up-front cost divided by that saving.