Module 6 — Quantization and Compression

At scale, the enemy is memory. A billion 768-dim float vectors is ~3 TB of RAM — impossible on one machine and expensive on many. Quantization shrinks vectors dramatically so they fit and so distance math runs faster. This module is what separates people who “use a vector database” from people who can run one at billion-scale on a budget.

Why bother? The memory math

One vector = dim × bytes_per_number. Standard embeddings use float32 = 4 bytes per number.

Vectors	Dim	float32 size	Fits where?
1M	768	~3 GB	Easy, one machine
100M	768	~300 GB	Big machine or cluster
1B	768	~3 TB	Cluster, or compress

Compression attacks bytes_per_number (use fewer bits) or the count of stored numbers (codes instead of values). Goals: fit more in RAM, search faster, pay less — while losing as little recall as possible.

The spectrum, from gentle to aggressive

1. Scalar Quantization (SQ) — “use smaller numbers”

The simplest idea: store each number in 8 bits (int8) instead of 32 bits (float32). You find the min and max value per dimension and map the float range onto 0–255. Result: 4× smaller, and modern CPUs do int8 math very fast.

• ✅ Simple, 4× memory cut, tiny recall loss (often <1%), fast. A great first step — frequently “free” quality.
• ❌ Only 4×; for billions you need more.

There’s also half-precision (float16 / bfloat16) — just 2× smaller with almost no loss, supported natively on GPUs. Often the easiest win of all.

2. Product Quantization (PQ) — “describe by nearest swatches”

Introduced in Module 5; here’s the full mechanism. PQ gives 8–32× compression.

Step by step:

1. Split each vector into m equal sub-vectors. A 768-dim vector with m=96 becomes 96 sub-vectors of 8 dims each.
2. Learn a codebook per sub-vector position: run k-means to find (typically) 256 representative sub-vectors (“centroids”) for that slice. 256 because the ID fits in exactly 1 byte.
3. Encode: replace each sub-vector with the ID (0–255) of its nearest centroid. Now the whole vector is just m bytes — for m=96, that’s 96 bytes instead of 3,072. A 32× reduction.
4. Search fast with ADC (Asymmetric Distance Computation): for a query, precompute the distance from each query sub-vector to all 256 centroids → a small lookup table. Then a stored vector’s distance is just m table lookups and adds. No full-precision math per vector. Blazing fast and memory-light.

The paint-swatch analogy again: rather than the exact RGB of every pixel, you say “this slice is closest to swatch #137, that slice to swatch #42…” You store swatch numbers, not exact colors. Slightly off, massively smaller.

• ✅ Huge compression (8–32×+), fast approximate distances, the backbone of billion-scale IVF-PQ.
• ❌ Lossy (recall drops), needs a training step to learn codebooks, and codebooks must be retrained if data distribution shifts.

3. OPQ (Optimized PQ) — “rotate first for a better fit”

PQ works best when the information is spread evenly across the sub-vector chunks. Real embeddings often have correlated dimensions, so some chunks carry more meaning than others. OPQ applies a learned rotation to the vectors before PQ so the variance is balanced across chunks. Same memory, better recall. If your database offers OPQ, it’s usually worth it.

4. Binary Quantization — “1 bit per number”

The most extreme common method: replace each number with a single bit — is it above or below zero? A 768-dim vector becomes 768 bits = 96 bytes → 32× smaller than float32 (96 bytes vs 3072); and distance becomes Hamming distance (count differing bits), which CPUs compute astonishingly fast with a couple of instructions.

• ✅ Insane speed and memory (up to 32×), great for a first-pass filter over enormous datasets.
• ❌ Lossy — but surprisingly, on high-quality high-dim embeddings (1024+), binary search can retain 90–95% of the quality, especially when followed by re-ranking. Some modern embedding models (e.g., from Cohere, Jina, Nomic) are explicitly trained to survive binarization.

The pattern that makes lossy compression safe: re-ranking (a.k.a. rescoring)

Here’s the trick that ties it all together and is used in nearly every serious system:

Two-stage search. Stage 1: use the compressed index (PQ or binary) to cheaply fetch a generous candidate set — say the top 200. Stage 2: re-score only those 200 candidates using the full-precision float32 vectors, and return the true top 10.

You get compression’s speed and memory for the 99.9% of vectors you ignore, and float32 accuracy for the handful you actually return. This is why “lossy” compression doesn’t have to mean “bad results.” Keep full-precision vectors on disk (cheap), keep compressed vectors in RAM (fast), re-rank from disk only the shortlist.

Variants you’ll see: binary first-pass → int8 rescore → float32 rescore (“rescoring cascades”), each stage cheaper-and-rougher feeding the next. Matryoshka embeddings (below) let one model serve multiple precisions for exactly this.

Matryoshka embeddings (MRL) — bonus modern trick

Some newer models (OpenAI text-embedding-3, Nomic, others) are trained so that the first N dimensions are themselves a usable, lower-quality embedding — like Russian nesting dolls. You can truncate a 3072-dim vector to its first 256 dims for a fast, tiny index, then use more dimensions only when re-ranking. One model, many size/quality trade-offs, no re-embedding. Very handy for cost control.

Putting numbers to it

For one 768-dim vector:

Representation	Bytes	Reduction	Typical recall impact
float32	3,072	1×	baseline (exact)
float16	1,536	2×	~none
int8 (SQ)	768	4×	very small
PQ (m=96)	96	32×	moderate, recoverable via re-rank
Binary	96	32×	larger, strongly recoverable via re-rank

(PQ and binary both land near 96 bytes here but via different mechanisms; combine PQ with IVF for the classic billion-scale recipe.)

Hands-on: scalar quantization, by hand

import numpy as np

def scalar_quantize(vectors):
    vmin, vmax = vectors.min(0), vectors.max(0)
    scale = (vmax - vmin) / 255.0
    codes = np.round((vectors - vmin) / scale).astype(np.uint8)  # 0..255
    return codes, vmin, scale            # 4x smaller than float32

def dequantize(codes, vmin, scale):
    return codes.astype(np.float32) * scale + vmin

v = np.random.randn(1000, 768).astype("float32")
codes, vmin, scale = scalar_quantize(v)
print("float32 bytes:", v.nbytes, "| int8 bytes:", codes.nbytes)  # 4x smaller
recon = dequantize(codes, vmin, scale)
print("avg abs error:", np.abs(v - recon).mean())                 # tiny

How databases expose this

You usually don’t implement quantization yourself — you turn it on:

• Qdrant: quantization_config with scalar / product / binary, plus optional on-disk originals for rescoring.
• Milvus: index types IVF_SQ8, IVF_PQ, SCANN, plus BIN_* for binary.
• FAISS: IndexIVFPQ, IndexScalarQuantizer, IndexBinaryFlat, IndexHNSWPQ, etc.
• Weaviate / Elasticsearch / pgvector: PQ / BQ / SQ options of varying maturity.

Your job is to choose the level, enable re-ranking, and measure recall (Module 11).

Gotchas

• Compressing without re-ranking and then being shocked at recall. Almost always pair lossy compression with a full-precision rescoring stage.
• Training codebooks on unrepresentative data. PQ/OPQ learn from a sample; if that sample doesn’t match production data, recall suffers.
• Distribution drift. As your data changes, old codebooks/clusters go stale. Periodically retrain/re-index.
• Over-compressing small datasets. If 3 GB fits in RAM, don’t bother with PQ — you’re trading accuracy for memory you didn’t need to save.
• Forgetting binary needs the right model. Binarization works well only on embeddings trained or suited for it (usually higher-dim). Test before trusting.

Check yourself

1. Why does memory, not just speed, force us to compress at billion scale?
2. Explain Product Quantization with the paint-swatch analogy.
3. What is the two-stage re-ranking pattern and why does it rescue lossy compression?
4. When is plain scalar (int8) quantization the smart, low-risk first move?
5. What problem does OPQ solve over plain PQ?

Answers: (1) A billion float32 vectors is ~3 TB of RAM — too large/expensive to hold; compression makes them fit and also speeds distance math. (2) Split the vector into chunks; for each chunk store the ID of the nearest entry in a small learned codebook (swatch book) instead of exact values — store swatch numbers, not colors. (3) Fetch a generous candidate set with the cheap compressed index, then re-score just those candidates with full-precision vectors; you get speed/memory on the ignored masses and exact accuracy on the few you return. (4) When you want an easy ~4× memory cut with near-zero recall loss and no big re-architecture. (5) PQ assumes information is balanced across chunks; OPQ rotates the data first so variance is spread evenly, improving recall at the same memory.

➡️ Next: 07-choosing-a-vector-database.md — the actual products and how to pick one.