Module 04 — Quantization

Goal of this module: Understand how numbers are stored in a model, why making them smaller (quantization) saves memory and often increases speed, what the trade-offs are, and which concrete methods (FP16/BF16, FP8, INT8, INT4, GPTQ, AWQ, GGUF, and KV-cache quantization) you should reach for and when. By the end you can look at a model and a GPU and decide a sensible precision strategy.

This is the first big lever for the “models are huge” problem from Module 00.

4.1 The core idea in one sentence

Quantization means storing the model’s numbers using fewer bits, trading a tiny bit of accuracy for big savings in memory and often higher speed.

That’s the whole concept. Everything else is detail about how to do it without hurting quality.

4.2 How numbers are stored: precision and bits

A model is millions/billions of numbers (parameters/weights). Each number is stored in a format that uses a certain number of bits. More bits = more precise (more decimal places, bigger range) but more space. The format is the parameter’s precision.

The common formats, from biggest to smallest:

Format	Bits	Bytes/param	Nickname	Role
FP32	32	4	”full precision” / “float32”	Training; rarely used to serve
FP16	16	2	”half precision” / “float16”	Common serving default
BF16	16	2	”bfloat16” / “brain float”	Like FP16 but wider range; very common
FP8	8	1	”float8”	Modern 8-bit, hardware-accelerated on newer GPUs
INT8	8	1	”8-bit integer”	Classic quantization
INT4	4	0.5	”4-bit”	Aggressive quantization, big savings

FP16 vs BF16 (a common question): both are 16-bit. FP16 spends its bits on more precision but a smaller range of values; BF16 keeps the same wide range as FP32 but with less precision. BF16 is often preferred because the wide range avoids numerical blow-ups, and modern GPUs support it natively. For serving purposes treat them as interchangeable “2-byte” baselines unless a model specifically requires one.

“Going from FP16 to INT8” halves the memory. “Going to INT4” quarters it. That is the headline benefit, and it maps straight onto the memory table from Module 01.

4.3 Why smaller numbers help — two distinct wins

Quantization helps for two separate reasons. Keep them straight.

Win 1 — Less memory (fits bigger models)

Fewer bytes per parameter means the weights take less VRAM. This is what lets a 70B model (≈140 GB in FP16) drop to ≈70 GB (INT8) or ≈35 GB (INT4) and suddenly fit on hardware it otherwise couldn’t. It also frees memory for a bigger KV cache, which means you can batch more users (Module 02/03) — a throughput win on top of the fit win.

Win 2 — More speed (because decode is memory-bound)

Recall the killer fact from Modules 01–02: decode is memory-bound — its speed is limited by how fast weights move through memory, not by compute. If each weight is half the bytes, you move half the data per token, so you can generate tokens roughly faster. Quantization doesn’t just shrink the model; it widens the effective hallway. This is why quantization is both a fit technique and a speed technique.

Subtlety: the speed win is real but not always exactly proportional, because (a) some operations are compute-bound and don’t benefit, (b) there is overhead to convert quantized numbers back and forth (“dequantization”), and (c) you only get the compute speedup if the GPU has hardware that natively does math in that format (e.g. FP8/INT8 Tensor Cores). On hardware without native low-precision support, you may get the memory win but a smaller speed win. Always check what your GPU accelerates (Module 01’s spec sheet skill).

4.4 Why it doesn’t destroy the model (the intuition)

It seems alarming to round off the numbers in a model — won’t it break? Usually not much, for a few reasons:

Neural networks are robust. They were trained with noise and have lots of redundancy. A little rounding is just a little more noise, which they tolerate well.
Smart quantization isn’t naïve rounding. Good methods (below) figure out which numbers matter most and protect them, and they pick rounding scales cleverly per group of weights rather than globally.
Not everything is quantized equally. Often weights are quantized aggressively while more sensitive parts stay higher precision.

Still, there is a cost: more aggressive quantization (especially INT4 and below) gradually degrades quality — the model may make slightly more mistakes, be a bit less nuanced, or occasionally produce worse outputs. The art is getting most of the memory/speed benefit while keeping quality loss negligible for your use case. Whether INT4 is “fine” or “too lossy” genuinely depends on the task — a casual chatbot tolerates more than a medical-coding assistant. Test on your own task; don’t trust a generic benchmark blindly.

4.5 Weight-only vs weight-and-activation quantization

A distinction that explains why there are so many methods.

Weight-only quantization: only the stored weights are compressed (e.g. to INT4), but the actual math is done after converting them back to a higher precision on the fly. This gives the big memory win and the memory-bandwidth speed win, and tends to preserve quality well. Most popular LLM quantization (GPTQ, AWQ, many GGUF modes) is weight-only.
Weight + activation quantization: both the weights and the activations (the live numbers flowing through the model as it runs) are quantized, so the math itself happens in low precision on Tensor Cores. This can give a bigger compute speedup, but it’s harder to keep accurate because activations sometimes contain extreme “outlier” values that low precision handles poorly. FP8 and INT8 schemes (like SmoothQuant, and FP8 on modern GPUs) live here.

You don’t need to memorize which scheme is which; you need to know the axis exists, so when a method says “W4A16” (weights 4-bit, activations 16-bit) you can read it. (WxAy = weights x-bit, activations y-bit.)

4.6 The named methods you’ll actually encounter

Here are the names you’ll see in model repositories and vLLM flags, each in plain terms.

GPTQ

A popular weight-only method (commonly 4-bit). It quantizes the weights carefully, one piece at a time, using a small calibration dataset to minimize the error introduced. Produces good-quality 4-bit models. Widely supported, including by vLLM. Think: “careful, calibration-based 4-bit weights.”

AWQ (Activation-aware Weight Quantization)

Another leading weight-only 4-bit method. Its insight: a small fraction of weights are far more important (because they multiply large activations), so it protects those important weights and quantizes the rest harder. Often excellent quality-for-size and fast. Also well-supported by vLLM. Think: “4-bit, but it shields the weights that matter most.”

GGUF (and the llama.cpp world)

GGUF is a file format (from the llama.cpp project) that packages a model with various quantization levels (you’ll see names like Q4_K_M, Q5_K_M, Q8_0 — higher number = more bits = better quality, bigger size). GGUF is especially popular for running models on CPUs and consumer hardware / Macs, and for mixed CPU+GPU setups. It’s the format of choice in the local/hobbyist ecosystem. Think: “the format for running models locally, often on a laptop, with flexible quant levels.” (vLLM is GPU-focused and primarily uses GPTQ/AWQ/FP8-style quantization; GGUF is more the llama.cpp/Ollama world, though support overlaps.)

FP8

An 8-bit floating-point format natively accelerated on the newest data-center GPUs. Because it’s done in hardware, FP8 can give both memory savings and genuine compute speedups with typically very small quality loss — making it a favorite for high-end production serving when the hardware supports it. Think: “modern 8-bit that the newest GPUs run natively and fast.”

INT8 (e.g. LLM.int8(), SmoothQuant)

Classic 8-bit integer quantization. LLM.int8() handles activation outliers by keeping them in higher precision; SmoothQuant “smooths” activations so both weights and activations quantize cleanly to INT8 for full compute speedup. Solid, well-understood, modest quality loss. Think: “the dependable 8-bit workhorse.”

A simple mental ranking

Quality (high → lower)         Memory savings (small → large)
FP16/BF16  (baseline)  ──────  none (baseline)
FP8 / INT8 (8-bit)     ──────  ~2× smaller
INT4 (GPTQ/AWQ)        ──────  ~4× smaller
even lower (3-bit, 2-bit) ───  ~5–8× smaller, quality drops faster

A common, safe default in 2025-era production: FP8 or INT8 if your hardware accelerates it and you want minimal quality risk; 4-bit AWQ/GPTQ when you need to fit a bigger model or maximize throughput and have validated quality on your task.

4.7 Don’t forget the KV cache: KV-cache quantization

A subtle but high-value technique that beginners overlook. Remember from Module 02 that the KV cache can consume as much memory as the weights, especially for long contexts and many concurrent users. So you can quantize that too:

KV-cache quantization: store the cached Keys and Values in 8-bit (or lower) instead of 16-bit.

This roughly halves KV-cache memory, which means you can hold more concurrent requests / longer contexts in the same GPU — directly boosting the batching that drives throughput (Module 02/03). vLLM supports FP8 KV cache, for example. The quality impact is usually small. When you’re memory-constrained on concurrency rather than on fitting the weights, KV-cache quantization is often the right lever — a distinction many people miss.

4.8 How quantized models are produced and obtained

You generally do not quantize from scratch yourself at first. Two common paths:

Download a pre-quantized model. The community publishes thousands of already-quantized models (e.g. “Llama-3-70B-Instruct-AWQ”, or various GGUF quants). You just point your serving engine at the right one. This is the fast path.
Quantize a model yourself using a toolkit (e.g. AutoGPTQ, AutoAWQ, llm-compressor, or built-in flows), feeding it a small calibration dataset (a few hundred representative text samples used to set the quantization scales well). You do this when no good pre-quantized version exists, or you want it tuned to your domain.

Either way, the key operational facts: a quantized model is just a different set of files; your serving engine must support that quantization method; and you should evaluate quality on your own task before trusting it in production.

4.9 A decision checklist

When deciding precision for a deployment, walk through this:

Does the FP16/BF16 model fit with room for KV cache and headroom? If yes and quality is paramount, you may not need to quantize at all. If no, quantization is likely mandatory.
What does my GPU natively accelerate? (FP8? INT8? — Module 01 spec sheet.) Prefer a format the hardware runs fast.
How sensitive is my task to quality? High-stakes → stay at 8-bit or test 4-bit very carefully. Tolerant tasks → 4-bit is often great.
Am I memory-limited by weights, or by concurrency/context? Weights → quantize weights. Concurrency/long context → also quantize the KV cache.
Validate. Run your real prompts through the quantized model and compare quality + measure the actual speed/throughput gain. Don’t assume; measure (Module 09).

Mental model to carry forward: Quantization is the lever that turns “this model doesn’t fit / is too slow / can’t batch enough users” into “now it does.” It interacts with everything: fit (Module 01), KV cache and batching (Module 02/03), and the optimizations and multi-GPU choices ahead (Modules 05/07). It is rarely the only lever you pull, but it’s almost always one of them.

Check your understanding

State what quantization is in one sentence, and name its two distinct benefits. (4.1, 4.3)
Given decode is memory-bound, explain why halving the bytes per weight tends to speed up decoding. (4.3)
What’s the difference between weight-only and weight-and-activation quantization, and why is the latter harder? (4.5)
In one phrase each, distinguish GPTQ, AWQ, GGUF, and FP8. (4.6)
When would KV-cache quantization help you even if the weights already fit comfortably? (4.7)
Why must you evaluate a quantized model on your own task rather than trusting a generic benchmark? (4.4, 4.8)

Key terms introduced

quantization · precision / bits · FP32 / FP16 / BF16 / FP8 / INT8 / INT4 · weight-only vs weight+activation quantization · WxAy notation · dequantization · GPTQ · AWQ · GGUF · SmoothQuant / LLM.int8() · calibration dataset · KV-cache quantization · pre-quantized model

Next: Module 05 — Inference Optimization Techniques, the toolbox of tricks — continuous batching, PagedAttention, speculative decoding, prefix caching, FlashAttention, and more — that make serving fast.