The Knowledge Problem
Module 02 — The Knowledge Problem
In Module 01 we learned the model’s knowledge lives in frozen, blurry, private weights. This module turns that into the specific problem RAG solves: hallucination, and the idea of two kinds of knowledge.
1. What is hallucination?
A hallucination is when a model produces something that sounds confident and fluent but is actually wrong or made-up. Importantly, the model isn’t “lying” — it has no concept of truth. Remember what it does: predict the next plausible token. A made-up but plausible-sounding answer is, to the model, just a good prediction.
The model is built to be fluent, not to be correct. When it doesn’t know something, it doesn’t go quiet — it generates the most likely-looking words, which can be confidently false.
Examples you’ll see in the wild:
- Inventing a citation, a legal case, or a research paper that doesn’t exist.
- Stating last quarter’s revenue with total confidence — and being completely wrong.
- Making up a feature your product doesn’t have.
- “Quoting” a policy that says the opposite of the real one.
2. Why hallucination happens — the blurry-photo analogy
Imagine you compressed every photo you’ve ever taken into one tiny file. Common scenes (your living room, your face) would still be recognizable. But a rare receipt photographed once? You’d recall it was a receipt, roughly — but the exact numbers? You’d unconsciously fill in plausible digits.
That’s exactly what a model does with rare or precise facts. Its weights store a lossy, compressed average of everything it saw. For frequently-repeated, common knowledge, the reconstruction is accurate. For rare details, exact figures, names, recent events, or anything it never saw, it fills in the most plausible blank — and that’s a hallucination.
Hallucination is therefore most dangerous exactly where businesses care most: specific, private, recent, precise facts.
3. The big idea: two kinds of knowledge
This single distinction is the backbone of everything in this tutorial.
Parametric knowledge
Parametric knowledge is what’s stored in the weights (remember, “parameters” = “weights”). It’s everything the model absorbed during training.
- ✅ Fast — it’s instantly available, no lookup.
- ✅ Free at use-time — already in the model.
- ✅ Fluent and broad.
- ❌ Frozen at the knowledge cutoff.
- ❌ Blurry — bad at exact/rare details.
- ❌ Can’t be cited — there’s no source to point to.
- ❌ Doesn’t include anything private or new.
Non-parametric knowledge
Non-parametric knowledge is information you give the model at the moment you ask, by putting it into the prompt. The model reads it fresh, right then.
- ✅ Always current — you supply today’s data.
- ✅ Exact — it’s the real text, not a memory.
- ✅ Citable — you know exactly which document it came from.
- ✅ Private/custom — it can be your documents.
- ✅ Swappable — change the document, change the answer instantly.
- ❌ Costs space in the context window (the model can only read so much at once).
- ❌ Requires you to find and fetch the right information first.
In one line: Parametric knowledge is what the model remembers. Non-parametric knowledge is what you show it right now.
4. Where RAG fits
Look at the lists above. Non-parametric knowledge is strong exactly where parametric knowledge is weak: freshness, exactness, privacy, citations. The catch is its one downside — you have to find and fetch the right information and fit it into the prompt.
That fetching-and-fitting problem is exactly what RAG automates.
- Retrieval = automatically finding the relevant documents for a question.
- Augmented = adding them into the prompt.
- Generation = the model writing an answer using them.
So Retrieval-Augmented Generation literally means: find the right text, add it to the prompt, then let the model generate an answer grounded in it. The next module makes this concrete.
5. The “context window” — why we can’t just paste everything
You might ask: why not just paste all our documents into the prompt every time?
Because the model can only read a limited amount at once — its context window, measured in tokens (Module 01). Even large windows can’t hold a company’s entire knowledge base, and even when they technically could, doing so is slow, expensive (you pay per token every single time), and the model gets worse at finding the needle when the haystack is huge.
So we need to be selective: fetch only the handful of passages actually relevant to this question. Being selective and accurate about that is the central engineering challenge of RAG — and most of this tutorial.
Check yourself
- Why is hallucination a prediction problem rather than a “lying” problem?
- Define parametric and non-parametric knowledge, and give one strength of each.
- Why can’t we solve the knowledge problem by pasting everything into the prompt?
- What do the three words in “Retrieval-Augmented Generation” each refer to?