Module 09 — Retrieval, Hybrid Search & Reranking

We can now find chunks by meaning (Module 08). But “find some relevant chunks” and “find exactly the right chunks and put the best one first” are different skill levels. This module covers the techniques that turn mediocre retrieval into great retrieval — the stuff that fixes most “the answers are just okay” complaints.

1. Hybrid search — meaning + keywords together

Recall the problem from Module 08: embedding search understands meaning but stumbles on exact terms (SKUs, error codes, names, acronyms). Meanwhile, old keyword search is the opposite — great at exact terms, clueless about meaning.

So combine them. Hybrid search runs both and merges the results:

Dense search = the embedding/vector search from Module 08 (“dense” because the vectors are full of numbers). Catches meaning and paraphrase.
Sparse search = keyword search, the best-known method being BM25 (a classic algorithm that ranks documents by how well their exact words match the query; “sparse” because it’s based on which specific words appear). Catches exact terms and rare tokens.

Best practice, near-universal: use hybrid search. Dense alone misses error codes; sparse alone misses paraphrases. Together they cover each other’s blind spots.

How do you merge two ranked lists? A simple, robust method is Reciprocal Rank Fusion (RRF): each result gets points based on how high it ranked in each list, and you sort by total points. You don’t need the formula — just know RRF is the standard “blend two result lists fairly” trick. (Weighted score blending is another option.)

2. Metadata filtering — narrow before you search

Before (or alongside) searching, filter by metadata (the tags from Module 07): date ranges, document type, language, and — critically — who’s allowed to see it.

Why it’s often the cheapest accuracy win: if you only search “this user’s documents, from this year, of type ‘policy’,” you’ve thrown away mountains of irrelevant candidates before ranking. Smaller, cleaner haystack → better top results, faster, and safer.

Combine filtering with hybrid search as your standard retrieval setup: filter down, then dense+sparse search the survivors.

3. Choosing K — how many chunks to fetch

K is how many chunks you retrieve. It’s a balance:

K too small (e.g. 1–2) → you might miss the chunk that actually holds the answer. The model can’t use what it never received.
K too large (e.g. 50 stuffed straight into the prompt) → you flood the model with mostly-irrelevant chunks (distractors). Counterintuitively this lowers answer quality — the relevant needle is buried in noise — and costs more tokens.

The pro pattern resolves this tension elegantly (next section): retrieve a generous K, then narrow it with reranking.

4. Reranking — the single highest-value upgrade

Here’s the most impactful technique in this whole module.

The retrieval in Module 08 is built for speed over millions of chunks, so it’s fast but a bit rough — it’s good at getting relevant chunks somewhere in the top 50, but not great at putting the single best one at position #1. A reranker fixes the ordering.

Why two different tools?

The embedding search uses a bi-encoder: it turns the query and each chunk into vectors separately, then compares. Fast (you pre-computed all chunk vectors), but a little lossy — it never looks at query and chunk together.
A reranker uses a cross-encoder: it reads the query and a candidate chunk together, as a pair, and judges relevance directly. Much more accurate — but too slow to run against millions of chunks.

So you use each for what it’s good at:

Retrieve broad, rerank precise, feed few.

Use fast vector+keyword search to get a generous shortlist (say top 50).

Use the slow-but-accurate reranker to carefully re-score those 50 and reorder them.

Keep only the best 3–8 and send those to the model.

This pattern — broad recall then precise reranking — fixes a huge share of “RAG gives okay-but-not-great answers” problems. If you add only one thing beyond basic retrieval, add a reranker.

5. Query transformation — fix the question before searching

Sometimes the user’s question is a poor search query even when it’s a clear question. A few techniques (deeper versions are in Module 11):

Query rewriting: clean up the question — fix typos, expand acronyms, add obvious synonyms — before searching.
Multi-query: generate a few rephrasings of the question, search each, and combine the results. Catches more relevant chunks when the wording is ambiguous.
Query decomposition: break a multi-part question (“compare A and B on price and warranty”) into sub-questions, retrieve for each, then combine.

These help most when questions are terse, jargon-heavy, or actually several questions in one.

6. Putting the retrieval stage together

A strong, standard retrieval pipeline looks like this:

User question
   │
   ├─ (optional) rewrite / expand the query
   │
   ├─ apply METADATA FILTERS (date, type, permissions)
   │
   ├─ HYBRID SEARCH: dense (vectors) + sparse (BM25), fused with RRF  → top 50
   │
   ├─ RERANK the 50 with a cross-encoder  → keep top 5
   │
   ▼
best 5 chunks → off to the prompt (Module 10)

Notice this is layered: each stage cleans up the previous one’s output. Filtering removes the clearly-irrelevant; hybrid search casts a meaning+keyword net; reranking sharpens the order; K-narrowing keeps the prompt clean.

Check yourself

Why does pure embedding (dense) search struggle with an error code like “ERR_4042,” and what fixes it?
What does hybrid search combine, and what is RRF for?
Explain “retrieve broad, rerank precise, feed few” and why a reranker is accurate but can’t be used on the whole library.
Why can a too-large K make answers worse?
What’s the cheapest way to shrink the haystack before ranking?

Next: Module 10 — Generation & Prompting →