Fine-Tuning vs. RAG vs. Prompting

The problem: a general model, a specific job

Foundation models are generalists. Your application needs a specialist: one that knows your products, follows your format, uses your tone and handles your edge cases. There are three main ways to close that gap:

Prompting — tell the model what to do, and show it, in the input.
Retrieval-augmented generation (RAG) — automatically fetch relevant information and add it to the input.
Fine-tuning — continue training the model on your own examples, changing its weights.

The most important insight is that these are not three levels of the same thing. They change different things.

What each one actually changes

	Prompting	RAG	Fine-tuning
Changes	The instructions	The knowledge available at answer time	The model's learned behavior
Best for	Task definition, format, tone, rules	Facts: private, large or frequently changing	Consistent style/format, narrow skills, smaller and cheaper models
Time to first result	Minutes	Days	Days to weeks
Data needed	A few examples	Your documents	Hundreds to thousands of high-quality examples
Updating	Edit the prompt	Update the documents	Retrain
Can cite sources	Only what's in the prompt	Yes	No

Prompting: always start here

Instructions, context and a few examples in the prompt. (See Prompt Engineering That Actually Works.)

Pros: instant iteration, no training, no extra infrastructure, works with the most capable models, easy to change.
Cons: limited by the context window; long prompts add cost and latency on every request (prompt caching, offered by many providers, reduces this); can't add large amounts of knowledge.

Modern models are strong instruction-followers, and a surprising number of projects never need anything more than a well-designed prompt plus good examples.

RAG: when the model needs knowledge it doesn't have

Search your documents for the passages relevant to each request and include them in the prompt. (See Build a RAG App From Scratch.)

Pros: handles huge and changing knowledge bases; updates are instant (just re-index a document); answers can cite sources, which builds trust and makes errors traceable; access control can be enforced per user.
Cons: more moving parts (parsing, chunking, embeddings, search); answer quality is capped by retrieval quality.

Choose RAG for: customer support over product docs, internal knowledge assistants, legal or policy Q&A, anything where facts change or must be verifiable.

Fine-tuning: when you need to change behavior

Train the model further on input → ideal-output examples, so the desired behavior becomes built in.

Pros: very consistent format and style; shorter prompts (the instructions are "baked in"); can make a small, cheap, fast model perform like a much larger one on a narrow task; teaches skills hard to describe in words.
Cons: needs a good dataset (the most underestimated cost); training and evaluation take time; every change means retraining; can degrade general abilities; not available for every model.

Modern fine-tuning is often done with parameter-efficient methods like LoRA, which train a small set of extra weights instead of the whole model — far cheaper, and practical on a single GPU for open-weight models. Some model providers also offer fine-tuning as a managed service.

Choose fine-tuning for: strict output formats at high volume, a distinctive brand voice, specialized classification or extraction, or cutting cost and latency by moving a well-defined task to a smaller model.

The biggest myth: "fine-tune it on our documents"

A very common plan is to fine-tune a model on company documents so it "knows" them. This usually disappoints:

Fine-tuning is good at teaching patterns and behavior, but unreliable at storing specific facts that the model can then recall accurately.
The model can't tell you where an answer came from, so you can't verify it.
When a document changes, the model is out of date until you retrain.
There's no way to stop one user from extracting information they shouldn't see.

Rule of thumb: RAG for knowledge, fine-tuning for behavior, prompting for everything you can.

Combining them

The approaches aren't exclusive — mature systems often use all three:

A prompt defines the task and rules.
RAG supplies up-to-date facts for each request.
A fine-tuned model answers in the exact format and voice required, or does the job at lower cost.

For example, a support assistant might use a small fine-tuned model that reliably produces the company's response format, with RAG pulling in the relevant help-center articles for each ticket.

A decision framework

Can a strong model do the task well with a good prompt and examples? Build a small test set and find out. If yes — ship it.
Are failures caused by missing or outdated knowledge? Add RAG.
Are failures about format, style or consistency that prompting can't fix — or is the task well-defined, high-volume and too expensive? Consider fine-tuning, once you have a test set to prove it helps.
Re-measure after every change. Each step adds cost and complexity; keep it only if it measurably improves results. (See Evaluating LLM Applications.)

Notice what drives every step: diagnosing why outputs fail. The fix for "it doesn't know our refund policy" (RAG) is completely different from the fix for "it ignores our response template" (better prompting, then perhaps fine-tuning).

Key takeaways

Prompting changes instructions, RAG changes available knowledge, fine-tuning changes behavior.
Start with prompting — it's fastest and often enough.
Use RAG for private, large or changing facts, especially when sources matter.
Use fine-tuning for consistent behavior or to make smaller models viable — not to store facts.
Combine them when needed, and let measured failures guide each step.