Module 02 — Why Fine-Tune? Prompting vs RAG vs Fine-Tuning

Goal: Learn the single most valuable judgment in this entire field — knowing when fine-tuning is the right tool and when it isn’t. Experts are separated from beginners less by their ability to run training jobs than by their ability to avoid running the wrong ones. We’ll build a clear decision framework you can defend in a meeting.

1. Three ways to customize a model

When a generic model isn’t doing what you want, you have three main levers. They are not competitors so much as a toolkit, and the best systems often combine them.

Lever 1 — Prompting (a.k.a. prompt/context engineering)

You change what you say to the model, not the model itself. This includes writing better instructions, giving examples in the prompt (“few-shot”), assigning a role (“You are a senior tax accountant…”), and structuring the request. The model’s weights are untouched.

Cost: Nearly free. Iterate in seconds.
Skill needed: Writing and experimentation.
Changes the model? No.

Lever 2 — RAG (Retrieval-Augmented Generation)

You give the model relevant information at the moment it answers. You store your documents in a searchable form, retrieve the pieces relevant to the user’s question, and paste them into the prompt as context. The model then answers using that fresh, specific information.

Cost: Moderate engineering (a search/retrieval system), low per-query.
Skill needed: Data plumbing, embeddings, search.
Changes the model? No — it changes the model’s inputs.

Lever 3 — Fine-tuning

You change the model’s weights by training it on examples, so the new behavior is baked in and doesn’t need to be re-explained every time.

Cost: Higher up-front (data + compute + skill). Often cheaper per query afterward because prompts get shorter.
Skill needed: This whole course.
Changes the model? Yes — permanently (in the saved copy).

2. The mental model that resolves 90% of confusion

People constantly ask “should I use RAG or fine-tuning?” as if it’s one-or-the-other. The cleanest way to think about it:

RAG gives the model new knowledge. Fine-tuning gives the model new behavior, skill, or style.

Need it to know something specific and changing (your docs, today’s inventory, a customer’s history)? → That’s a knowledge problem → RAG (or just put it in the prompt).

Need it to act a certain way reliably (a tone, a format, a reasoning pattern, a narrow task done consistently)? → That’s a behavior/skill problem → fine-tuning.

A sharper way to say it, popular among practitioners:

Fine-tuning is for the form of the answer; RAG is for the facts in the answer.

Two quick examples to lock it in:

“Make the model answer using our internal HR policy documents.” The policies are facts that change and there are too many to memorize reliably → RAG.
“Make the model always reply in our brand’s terse, friendly style and output valid JSON in our schema.” That’s a consistent behavior → fine-tuning.

And often: both. Fine-tune the model to follow your format and reasoning style, and use RAG to feed it the current facts. They compose beautifully.

3. What fine-tuning is genuinely good at

Reach for fine-tuning when you need:

A consistent style, tone, or persona. Brand voice, a character, a specific register (formal legal, casual support). Prompts can suggest tone; fine-tuning makes it automatic and reliable.
A specific, narrow task done very reliably. Classifying tickets, extracting fields, converting English to your SQL dialect, summarizing in your exact format. Fine-tuning can make a small model match a much larger one on a narrow task.
A particular output format every time. Always valid JSON, always your template, always the same structure — without a giant fragile prompt.
Implicit skills hard to describe in words. “Write like this author,” “diagnose like our best support agent.” Easier to show with examples than to explain with instructions.
Latency and cost reduction. If your perfect prompt is 2,000 tokens of instructions and examples, fine-tuning can bake that in so your prompt drops to 50 tokens — faster and cheaper at scale.
Using a smaller, cheaper, self-hosted model. Fine-tune an 8B model to do one job as well as a frontier model does it generally, then run it cheaply and privately on your own hardware. This is one of the biggest commercial drivers.
New formats or modalities the base model handles poorly (e.g., a function-calling format, a domain-specific markup).

4. What fine-tuning is bad at (read this twice)

Beginners reach for fine-tuning to solve problems it can’t solve, waste a week, and conclude “fine-tuning doesn’t work.” Avoid these traps:

Injecting large, changing factual knowledge. Fine-tuning is a poor, expensive, lossy way to make a model “know” your 10,000 documents. The facts get smeared across weights, the model still hallucinates details, and the moment a document changes you’d have to retrain. Use RAG for knowledge.
Facts that must be exactly right and up to date. Prices, inventory, policies, this quarter’s numbers. The model will confidently mangle them. RAG or tool-calling, not fine-tuning.
Fixing problems a better prompt would fix. Always try prompting first. It’s free and instant. An astonishing number of “we need to fine-tune” situations are solved by a clearer prompt or a couple of examples.
When you have almost no data. Fine-tuning needs examples (often hundreds to thousands of good ones). With ten examples, put them in the prompt as few-shot instead.
Adding brand-new reasoning ability it fundamentally lacks. Fine-tuning shapes and surfaces existing capabilities far more than it creates wholly new ones. You can teach a model your format of reasoning; you generally can’t teach a weak model to suddenly be a strong reasoner from a few thousand examples.

The expert’s reflex: Prompt first. Add retrieval if the problem is knowledge. Fine-tune when the problem is behavior, consistency, format, cost, or running a smaller model — and you have data to show it what you want.

5. The decision framework (use this in real life)

Walk top to bottom. Stop at the first “yes.”

START: The model isn't doing what I want.
│
├─ Q1: Have I seriously tried improving the PROMPT
│      (clear instructions, a few examples, a role)?
│      └─ No  → Do that first. It's free. Come back if still stuck.
│      └─ Yes → continue
│
├─ Q2: Is the core problem that the model LACKS SPECIFIC FACTS/KNOWLEDGE
│      (my documents, current data, user-specific info)?
│      └─ Yes → Use RAG (or put the facts in the prompt). Not fine-tuning.
│      └─ No  → continue
│
├─ Q3: Is the problem about consistent BEHAVIOR, STYLE, FORMAT, or a
│      NARROW TASK done reliably — OR about cost/latency/using a smaller
│      self-hosted model?
│      └─ No  → Re-examine the problem; you may not need any of these.
│      └─ Yes → continue
│
├─ Q4: Do I have (or can I create) at least a few hundred good
│      input→output EXAMPLES of the desired behavior?
│      └─ No  → Get/generate data first (Modules 09–10), or use few-shot
│               prompting for now.
│      └─ Yes → FINE-TUNE. (And consider combining with RAG.)

Keep this diagram. In job interviews and design reviews, being able to reason through it out loud marks you as someone who understands the field rather than someone chasing the shiniest technique.

6. Cost and effort, compared honestly

Dimension	Prompting	RAG	Fine-tuning
Up-front effort	Minutes	Days–weeks (build retrieval)	Days (data + training)
Per-query cost	Higher (long prompts)	Medium	Often lower (short prompts)
Keeps facts current	Manual	Yes, easily	No (retrain to update)
Bakes in behavior/style	Weakly	No	Yes, strongly
Needs training data	No	No (needs documents)	Yes
Needs ML skill	Low	Medium	Higher (this course)
Lets you use a small/private model	Limited	Helps	Strongly
Risk of hurting other abilities	None	None	Yes (“forgetting” — Module 05)

There’s a hidden cost of fine-tuning worth flagging now: a fine-tuned model can get worse at things you didn’t train on (catastrophic forgetting), and you take on the ongoing burden of versioning, evaluating, and maintaining a model. RAG and prompting carry none of that maintenance weight. Factor it in.

7. Real-world scenarios — you decide (then check)

Try each before reading the answer.

A. A law firm wants the model to answer questions grounded in their library of 50,000 case documents, always citing the source. → RAG. This is changing knowledge that must be cited exactly. Fine-tuning would blur it and can’t cite.

B. A startup wants every chatbot reply to match their quirky, super-concise brand voice and never exceed two sentences. → Fine-tuning. Pure behavior/style consistency. (A prompt helps, but fine-tuning makes it bulletproof and cheaper.)

C. A team’s 2,500-token mega-prompt works great but is slow and expensive at a million calls/day. → Fine-tuning to bake the instructions in and shrink the prompt — a classic cost-reduction win.

D. A solo developer wants the model to use their company’s internal Python library correctly; the library has good docs and changes monthly. → RAG (feed the current docs). It changes too often and is too factual for fine-tuning.

E. A company wants to replace expensive frontier-model API calls for ticket classification with a cheap model they host themselves, hitting 95%+ accuracy on their categories. → Fine-tuning a small model. Narrow task + cost/privacy goal = textbook fit.

F. A medical-notes tool must (1) follow a strict structured format AND (2) reference the patient’s specific record. → Both. Fine-tune for the format/behavior; RAG for the patient-specific facts.

If you got most of these, you already have the most valuable instinct in the field. Everything from here is execution.

Module 02 checklist

I can state “RAG = knowledge, fine-tuning = behavior” and give an example of each.
I can list three things fine-tuning is great at and three it’s bad at.
I can walk the decision framework out loud.
I understand fine-tuning’s hidden costs (forgetting, maintenance).
I correctly classified the scenarios above.

➡️ Next: Module 03 — Training Fundamentals