RAG vs Fine-Tuning vs Prompt Engineering: When to Use Each (2026 Guide)

The customer says "we need to fine-tune a model on our data." They might be right — but you don't know that yet, and neither do they. Your job is to test the proposal with evidence before anyone burns six weeks and a training budget.

The real question: what is failing — knowledge, behavior, or both?

Dev's request bundles three different needs into one proposed solution. Untangle them before you pick a technology:

• Knowledge: "The model doesn't know our procedures." The gap is information — facts that live in the manual, not in the model's training data.
• Behavior: "We want answers in our voice." The gap is how it responds — tone, format, structure, consistency.
• Freshness: "The manual updates every quarter." The gap is time — whatever you build must track a moving target.

Each gap points at a different tool. That's the whole decision, and it's why "should we fine-tune" is the wrong first question. The right first question is: which gap are we actually trying to close?

First, what an LLM actually does: training vs. inference

An LLM's life has two phases, and the three approaches split across them:

• Training is when the model's weights are set — expensive, slow, done rarely. This is where the base model learned language, and where fine-tuning would continue the process on your examples.
• Inference is when the model runs — every API call you make. The weights don't change; the model just reads your input and generates output.

Prompting and RAG live entirely in inference. Nothing about the model changes between your calls — you're either wording the request better or attaching better context to it. Fine-tuning is the only option here that re-enters training — which is exactly why it's the slowest to iterate, the hardest to roll back, and the one you should reach for last, not first.

The three approaches, defined in one paragraph each

Prompt engineering (the API path). You call a hosted model and get better behavior through instructions, examples, and structured output. No weights are updated. Whether anything you send is stored or retained depends on the provider's policies and your deployment's configuration — that's a privacy decision to make explicitly, not a technical guarantee. Your leverage is how precisely you specify the task. Covered in Calling LLM APIs in Production and Prompt & System Design.

RAG (retrieval-augmented generation). You keep the model frozen and give it relevant external information at query time: retrieve the right passages, place them in the prompt, generate the answer grounded in them. The model's underlying knowledge doesn't change — but its answers can draw on your documents, including ones written yesterday. Covered in RAG Systems End-to-End — including the part everyone skips, document ingestion.

Fine-tuning. You continue training the model's weights on your own examples so its behavior changes: tone, format, domain reasoning patterns, task-specific accuracy. The knowledge cutoff doesn't move the way people think it does — fine-tuning teaches the model how to respond far more reliably than what is true.

What actually changes: a visual

If all three approaches "use an LLM," what's different under the hood? Follow what flows into the model and what comes out changed:

Prompting changes the input. RAG supplies retrieved information at request time. Fine-tuning updates model parameters during training. Same model, three completely different theories of where the improvement comes from.

One driver question, three implementations

Let's make it concrete with a single question Dev's drivers actually ask:

Approach A — prompt engineering. The model gets instructions like "Answer concisely and list the next three steps." It follows the format beautifully — but without access to the company manual, it doesn't know the actual breakdown procedure. It may invent plausible-sounding steps that contradict company policy. Format: excellent. Facts: unreliable.

Approach B — RAG. The system retrieves the relevant manual passage — say, "Secure the vehicle, contact dispatch, and open an incident ticket" — and the model answers from that passage, citing the section. The answer is grounded in the company's actual procedure, and when the procedure changes next quarter, updating the document updates every future answer.

Approach C — fine-tuning. The model trains on approved Q&A examples, so it consistently responds in the company's preferred tone and structure. But fine-tuning alone doesn't guarantee it knows the current breakdown procedure — it learned whatever the training examples contained, frozen at training time.

The key insight most teams miss: RAG and fine-tuning are not mutually exclusive. A company might use RAG for current procedures and fine-tuning for a consistent response style — retrieval keeps the facts fresh, training keeps the voice consistent. Combine methods when the evals justify each one independently, which is exactly what the evaluation section below is for.

The decision flowchart

Start with the beginner version. Every branch is a question about the problem, not the technology:

Now the detailed engineering version — the one to walk through with Dev. Note the two branches beginners miss: fast-changing information (which punishes anything frozen, including weights) and the combination exit:

Notice the shape: most exits point back to cheaper options or to combinations. That's the base rate — most "we need fine-tuning" requests resolve into RAG, prompting, or a deliberate combination once someone asks what the model is actually failing at.

When each one wins

When each one breaks

Every approach has a characteristic failure mode. The customer will describe the symptom; you diagnose the cause.

Details that bite in production

A few realities beginners don't expect — each has ended a real deployment:

• Document permissions. RAG must respect access control: the retriever must never surface documents the asking user isn't authorized to see. Retrieval over a shared corpus without per-user filtering is a data leak with good latency.
• Data privacy. Private documents sent to a hosted API, and private examples used for fine-tuning, need explicit retention decisions — what the provider stores, for how long, and under what terms. Get this in writing before the pilot, not after.
• Versioning. Prompts, retriever configurations, and model snapshots all change. Version each one, and re-run evals on every change — "we only tweaked the prompt" has broken more demos than any model upgrade.
• How RAG actually finds passages. Retrieval typically uses embeddings and vector search — sometimes combined with keyword or hybrid search. You don't need the math yet; you need to know that retrieval quality is a system you build and measure, not a library you import.

What a fine-tuning dataset actually looks like

Fine-tuning consumes examples, not documents. A single training example is an input-output pair showing the behavior you want:

{
  "messages": [
    {
      "role": "user",
      "content": "My delivery will be late."
    },
    {
      "role": "assistant",
      "content": "I understand. Please notify dispatch and provide your updated ETA."
    }
  ]
}

This is an illustrative training example of a desired response pattern — not a complete dataset, and not a guarantee the behavior will be learned. Real fine-tuning needs hundreds of such pairs, and one rule is non-negotiable: your training examples and your evaluation questions must be separate sets. Grading a model on the examples it trained on tells you it can memorize, not that it can generalize. The eval set from the next section is the honest judge.

The cost and operations dimension

Technical merit isn't the whole decision. Each path has a different operational signature — what it costs to build, what it costs to run, and what rots over time. The sketches below are illustrative order-of-magnitude comparisons, not measurements from a real deployment; your workload's evals are the only numbers that count.

How to actually decide: evals, not opinions

Dev is convinced fine-tuning is the answer. You're fairly sure RAG is. Neither of you should win this argument with opinions. Build a small eval set first: 30–50 real questions from Dev's drivers, with known-good answers from the operations manual. Then:

1. Baseline: run the eval set against plain prompting. Record the score.
2. RAG: build the minimal retrieval pipeline, run the same set. Record the score.
3. Compare honestly. If RAG beats prompting on the questions that matter — and failure analysis shows retrieval (not reasoning) as the bottleneck — you have your answer, with evidence Dev can take to his leadership.
4. Fine-tune only if the evals show a behavior gap that retrieval can't close and you can name what the fine-tune must change. "Make it better" is not a training objective.

What the results look like

Here's the kind of scorecard that settles the argument — illustrative scores for teaching, not measured benchmarks:

Read it like an FDE: RAG dominates on questions needing proprietary manual facts and on evidence — the fine-tuned model answers in the right format (39/40) but can't cite what it was never trained to retrieve. That supports choosing RAG for the pilot. It does not establish that RAG is universally superior — on a workload where format consistency is the whole job, the fine-tuned column would tell a different story. The scorecard answers "which approach for this workload," nothing more.

This is the Evals & Tracing discipline applied at decision time: the rubric defines correctness before anyone trains anything, and the numbers — not the loudest voice — pick the architecture.

What to tell the customer

Back on the call with Dev. Here's the honest version of the conversation:

That conversation is the job. The technology choice was the easy part — the hard part was refusing to let a confident customer buy six weeks of the wrong work.

CityOps tie-in

You've already lived this decision. In Spine Milestone 4: Ask CityOps, the 311 assistant answers from city procedures — that's the RAG path, and the milestone's eval set is exactly the "prove it first" instrument from this post. If you revisit that milestone now, ask yourself: which of its questions would fine-tuning have answered better? (Spoiler from the field: when the procedures updated mid-project, the teams with retrieval updated a document; the teams imagining fine-tuning would have retrained.)