Module 03 — What RAG Is

You now know the problem (frozen, blurry, private weights → hallucination) and the fix in principle (hand the model real text at question time). This module makes RAG concrete with a story, a step-by-step walkthrough, and the exact vocabulary.

1. The librarian analogy

Imagine a brilliant but forgetful professor. They’ve read enormously, so they speak fluently on almost anything — but they finished all their reading years ago, they mix up exact details, and they’ve never seen your company’s files.

Now give that professor a research assistant. When you ask a question, the assistant dashes into the library, finds the three most relevant pages, and slides them across the desk. The professor reads those pages and answers based on them — accurately, with the source right there to point at.

The professor = the language model (great at reasoning and writing, bad at recall of specifics).
The research assistant = the retrieval system.
The library = your collection of documents (the knowledge base).
Sliding the pages over = augmenting the prompt.
The professor reading and answering = generation.

RAG is giving the model a research assistant. That’s the whole concept.

2. RAG step by step (what actually happens when you ask a question)

Say you ask a RAG-powered support bot: “How long do I have to return a laptop?”

You ask a question. Your text is the query.
The system searches the knowledge base for passages related to your query. (How searching-by-meaning works is Module 08 — for now, trust that it can find relevant text.)
It pulls back the top few relevant passages — say, three paragraphs from the returns policy.

It builds a new prompt behind the scenes that looks roughly like:

Use ONLY the context below to answer. If the answer isn't there, say you don't know.

Context:
[Paragraph 1 from the returns policy...]
[Paragraph 2...]
[Paragraph 3...]

Question: How long do I have to return a laptop?

The model reads that prompt and generates an answer — grounded in the real policy text, e.g. “You have 30 days to return a laptop, per the returns policy.”
(Optionally) it cites the source, so you can verify.

Notice: the model never relied on its blurry memory. It transcribed and reasoned over text it was just handed. Update the policy file, and tomorrow the same question gives the new, correct answer — no retraining.

3. The two halves of a RAG system

Every RAG system has two distinct phases. Keep them separate in your head — it’ll save you endless confusion later.

Phase A — Indexing (done ahead of time, in the background)

This is preparing the library so it’s searchable. You take your documents, break them into pieces, convert each piece into a searchable form, and store them. This happens before any user asks anything, and gets refreshed as documents change. (Modules 07–08.)

Phase B — Retrieval + Generation (done live, per question)

This is the part in Section 2 above: for each question, search the prepared library, fetch the best pieces, and generate a grounded answer. (Modules 09–10.)

Analogy: Indexing is organizing and cataloguing the library. Retrieval+Generation is the assistant running in to grab pages when a question comes. You catalogue once (and re-catalogue as books change); you fetch on every question.

4. What RAG gives you (the benefits, recapped)

Freshness — answers reflect whatever’s in the library right now; update a file, not a model.
Accuracy / less hallucination — the model works from real text, not foggy memory.
Citations / trust — you can show exactly where each answer came from.
Privacy / customization — the library can be your private documents.
Access control — different users can be allowed to retrieve different documents.
Cost-efficiency — you fetch only the relevant slice instead of stuffing everything in.

5. What RAG does not do (setting expectations)

RAG is powerful but it’s not magic, and misunderstanding its limits is how projects go wrong:

It doesn’t make the model smarter or change its writing style — that’s a different tool (fine-tuning, Module 04).
It can’t answer from documents it didn’t retrieve. If the assistant grabs the wrong pages, the answer suffers — “garbage in, garbage out.” Most RAG failures are really retrieval failures.
It doesn’t automatically do math or multi-step reasoning over your data — it answers from fetched text.
It’s only as good as your library and your search. A messy, outdated, or poorly-searched knowledge base gives messy answers.

This is why the bulk of this tutorial is about doing retrieval well. The model is rarely the bottleneck; the finding-the-right-text part is.

Check yourself

Map each part of the librarian analogy to a real RAG component.
What are the two phases of a RAG system, and how often does each run?
Why does “garbage in, garbage out” apply so strongly to RAG?
Name three benefits RAG provides that a raw model can’t.

Next: Module 04 — RAG vs Training vs Fine-Tuning vs Prompting →