Module 16 — Learning Roadmap & Capstone

Reading gets you understanding; building gets you expertise. This final module is your hands-on path from “I understand RAG” to “I can architect and advise on RAG.” It’s structured as stages — each with a concrete thing to build and a way to know you’ve got it.

You don’t have to be a strong programmer to benefit, but doing even a simple build cements the concepts far better than reading alone.

Stage 0 — Mental model (½ day) ✅ you’ve basically done this

Goal: explain RAG, and when not to use it, to a non-technical executive in two minutes. Do: without looking, write a one-paragraph explanation covering: what weights are, why models hallucinate, what RAG does, and the knowledge-vs-behavior distinction. You’ve got it when: you can field “why not just fine-tune on our docs?” instantly and correctly.

Stage 1 — Build a naive RAG by hand (1–2 days)

Goal: feel every moving part with nothing hidden. Do: from scratch, without a big framework:

Take ~20 documents you care about.
Chunk them (start with fixed-size + overlap).
Embed each chunk with an embedding model.
Store the vectors (a simple library like FAISS, or pgvector).
For a question: embed it, retrieve the top-K nearest chunks.
Stuff them into a grounded prompt and generate an answer.

Why no framework first: tools like LangChain/LlamaIndex hide exactly the concepts you’re trying to learn. Build it raw once; then a framework is a convenience instead of a black box. You’ve got it when: you can point at each line and say which module it implements.

Stage 2 — Build the way to measure, then watch it fail (2–3 days)

Goal: the professional mindset shift from “looks good” to “measured.” Do:

Write a golden dataset: 50 real questions + correct answers + which chunk holds each answer.
Implement Recall@K (did the right chunk get retrieved?) and a faithfulness check (is the answer supported by the context? — LLM-as-judge is fine).
Run your Stage 1 system against it. Watch it fail on edge cases.

You’ve got it when: you can state your system’s Recall@K and faithfulness as numbers, and point to specific failures.

Stage 3 — Earn each upgrade (about a week)

Goal: learn which improvements matter for your data, by proof not faith. Do: add improvements one at a time, re-measuring after each:

chunk overlap / better (structural) chunking
hybrid search (add BM25 to your vector search)
a reranker
metadata filtering
query rewriting
parent-document (small-to-big) retrieval

Keep a little table: metric before → after for each change. Some will help a lot, some barely — that surprise is the lesson. You’ve got it when: you can say “for this corpus, the reranker added X to Recall and faithfulness; multi-query didn’t help” — backed by numbers.

Stage 4 — Production concerns (about a week)

Goal: make it real-world-safe. Do: add and reason about:

incremental indexing (handle a changed and a deleted document correctly)
access control by metadata (user A can’t retrieve user B’s docs)
a prompt-injection test (put a malicious instruction in a document; confirm it doesn’t hijack the system)
prompt/semantic caching
latency and cost measurement (p95, cost per query)

You’ve got it when: you can walk through the Module 14 production checklist and demonstrate each item.

Stage 5 — Advanced patterns on demand (ongoing)

Goal: know the higher rungs of the ladder and when to climb them. Do: when (and only when) your eval reveals a gap, implement the matching pattern: HyDE or multi-query (query/doc mismatch), decomposition (multi-part questions), RAPTOR (global “summarize the topic” questions), GraphRAG (relationship/multi-hop), agentic or corrective RAG (poor or multi-step retrieval), multimodal (image/scan-heavy corpora). You’ve got it when: for a given failure, you can name the one pattern that addresses it — and justify the added cost.

Stage 6 — Architect fluency (ongoing)

Goal: become the person who decides, not just builds. Do: take real (or invented) briefs and run them through the Module 15 frameworks. Practice decomposing a request: “this part is RAG, this part is SQL, this part is fine-tuning, and honestly this part doesn’t need AI at all.” You’ve got it when: your most common advice includes “you don’t need RAG here, and here’s why” — and you can defend the RAG-vs-long-context call for any scenario.

The capstone project

Build a support assistant over a real document set (your own docs, an open dataset, or a product’s public help center) that demonstrates the full journey:

A clean indexing pipeline (parse → chunk → embed → store with metadata) that handles updates and deletes.
Hybrid retrieval + reranking with metadata filtering.
Grounded, cited generation with an honest “I don’t know.”
A golden eval set and reported retrieval + generation metrics.
Access control so users only see permitted documents.
A short write-up: which design choices you made, what your metrics are, what you’d add next and why — and an explicit note on where RAG was and wasn’t the right tool.

That write-up is essentially a solution-architecture document. If you can produce it and defend every decision with a measured reason, you can advise solution architects on RAG. That was the goal.

Where to keep learning

Re-read Modules 12 (Evaluation) and 15 (Frameworks) periodically — they’re the ones experts return to.
Follow the evolving tooling (embedding-model leaderboards like MTEB, vector databases, eval frameworks like RAGAS/Phoenix) — the concepts in this tutorial are durable, but specific products move fast, so always validate current specifics before recommending a stack.
Teach it. Explaining RAG to someone else (using these modules) is the fastest way to find the gaps in your own understanding.

Final check yourself

Could you build the Stage 1 naive RAG today, and name the module behind each step?
Could you produce a golden eval set for a corpus you care about?
Given a vague business request, could you decide RAG vs fine-tune vs SQL vs long-context — and explain it?
Could you write the capstone’s architecture write-up?

If yes — you’re there. Go advise some architects.

Back to: Start Here · Glossary