Module 05 — Inference Optimization Techniques

Goal of this module: Build your toolbox. These are the core techniques that make modern LLM serving fast and cheap: continuous batching, PagedAttention, FlashAttention, prefix caching, chunked prefill, speculative decoding, disaggregated prefill/decode, and more. For each, you’ll learn what problem it solves, how it works in plain terms, and when it helps. This is the conceptual payload that Module 06 (vLLM) then puts into practice.

Keep Module 02’s question in your pocket for each technique: is this attacking prefill or decode? saving compute or memory?

5.1 Map of the toolbox

Optimizations fall into a few families. Here’s the landscape before the details:

THROUGHPUT (serve more users per GPU)
  • Continuous batching      — never let the GPU idle between requests
  • PagedAttention           — pack KV cache memory efficiently → bigger batches
  • Prefix caching           — reuse work across requests that share a prefix
  • Disaggregated prefill/decode — stop the two phases from interfering

LATENCY (make each request faster)
  • FlashAttention           — faster, memory-light attention kernels
  • Chunked prefill          — keep long prefills from stalling everyone's decode
  • Speculative decoding     — generate several tokens per step
  • CUDA graphs / kernel fusion — cut per-step overhead

MEMORY (fit more in VRAM → enables the above)
  • PagedAttention           — eliminate KV-cache waste
  • Quantization (Module 04) — smaller weights & KV cache
  • KV-cache offloading      — spill cold cache to CPU/disk

Many techniques help on more than one axis. Let’s go through them.

5.2 Continuous batching (the throughput superpower)

Problem it solves: Recall batching from Module 02 — running many requests together is how you turn idle, memory-bound decode into useful throughput. But there’s a naïve way to batch that wastes most of the benefit.

The naïve way — “static batching”: collect, say, 8 requests, run them together until all 8 finish, then start the next 8. The problem: requests finish at different times (one wants 10 tokens, another wants 500). With static batching, the GPU slot for the request that finished at token 10 sits empty for the remaining 490 steps, waiting for the slowest one. Lots of wasted capacity, and new requests wait for the whole batch to clear.

Continuous batching (a.k.a. in-flight batching, dynamic batching) fixes this: the scheduler works at the granularity of a single token step. After every step, it can remove finished requests and slot in new waiting ones, on the fly. The batch is constantly refreshed. No request waits for the whole batch; no GPU slot sits idle while others continue.

Static batching:                 Continuous batching:
[req A ▓▓▓▓▓▓▓▓▓▓ done]           [A ▓▓▓ done][E ▓▓▓▓▓▓...]
[req B ▓▓ done....idle.....]      [B ▓ done][F ▓▓▓▓▓▓▓▓...]
[req C ▓▓▓▓▓▓ done..idle...]      [C ▓▓▓▓ done][G ▓▓▓...]
[req D ▓▓▓▓▓▓▓▓▓▓ done]           [D ▓▓▓▓▓▓ done][H ▓▓...]
   ↑ slots idle, waiting          ↑ slots refilled every step

Impact: this is one of the largest throughput wins available — often several times the throughput of static batching at the same latency. It’s a foundational feature of every serious modern engine (vLLM, TGI, TensorRT-LLM all do it). If you remember one technique from this module, remember this one. (Attacks: decode; saves: wasted compute/idle time.)

5.3 PagedAttention (vLLM’s signature idea)

Problem it solves: To batch many requests you need all their KV caches in GPU memory (Module 02). The classic approach reserved a contiguous block of memory for each request, sized for its maximum possible length. That’s terribly wasteful:

A request that might generate 2,000 tokens but actually generates 50 still reserved space for 2,000 → huge internal waste.
Different reserved blocks leave unusable gaps → fragmentation.
Result: you run out of KV memory and can’t batch as many users as the GPU could actually hold. Reported waste in early systems was on the order of 60–80% of KV memory.

The insight — borrow from operating systems. Decades ago, OS designers solved the same “programs need variable, growing memory without waste” problem with virtual memory / paging: chop memory into small fixed-size pages and hand them out on demand, keeping a table that maps “logical position” → “physical page.” The pages don’t need to be next to each other.

PagedAttention does exactly this for the KV cache:

The KV cache is split into small fixed-size blocks (pages), each holding the K/V for a handful of tokens.
A request gets blocks allocated on demand as it generates, not reserved up front.
A per-request block table maps the sequence’s logical token positions to wherever the physical blocks actually live — which can be scattered anywhere in memory.

Payoff:

Almost no waste (a request only holds blocks it’s actually using; waste drops to a few percent).
No fragmentation (fixed-size blocks fit anywhere).
Because memory is used so efficiently, you can fit many more concurrent requests’ KV caches → much larger batches → much higher throughput (the original vLLM paper reported 2–4× over prior systems).
Bonus — sharing: because blocks are indirected through a table, two requests can share identical blocks. Several outputs from the same prompt (e.g. multiple samples, or beam search) can share the prompt’s KV blocks with copy-on-write, saving even more memory. This sharing is also what makes prefix caching (next) clean to implement.

PagedAttention is the reason vLLM launched to such impact: it directly attacks the memory bottleneck that limits batching, which (via Module 02’s chain) limits throughput. (Attacks: KV memory; enables: bigger batches → throughput.)

5.4 Prefix caching (don’t redo shared work)

Problem it solves: Many requests share the same beginning. A chatbot has a long system prompt prepended to every conversation. A RAG app stuffs the same document into many queries. A multi-turn chat re-sends the whole history each turn. Re-running prefill over that identical prefix every time is wasted compute (and worsens TTFT).

Prefix caching (a.k.a. automatic prefix caching, APC): keep the KV cache for common prefixes around and reuse it. When a new request arrives whose beginning matches a cached prefix, the engine skips re-computing that part’s KV cache and jumps ahead — only the new portion needs prefill.

Impact: dramatically lower TTFT and less prefill compute for workloads with shared prefixes (system prompts, few-shot examples, long documents, multi-turn chat). The savings scale with how much text is shared. PagedAttention’s block-sharing machinery makes this natural to implement: matching prefixes literally point at the same KV blocks. (Attacks: prefill; saves: redundant compute → better TTFT + throughput.)

When it does not help: workloads where every prompt is unique with no shared prefix get little benefit. Know your workload.

5.5 FlashAttention (a faster attention kernel)

Problem it solves: The attention computation (Module 02) naively creates a large intermediate table comparing every token to every other token, and shuffles it back and forth to the slow VRAM tier (Module 01’s memory hierarchy). That memory traffic makes attention slow and memory-hungry, especially for long sequences.

FlashAttention: a cleverly engineered kernel that computes attention without ever writing that giant intermediate table to slow memory. It keeps the work in the fast on-chip memory tiers (registers/SRAM), processing attention in tiles and combining results as it goes (“online softmax”). Same mathematical result, far less memory movement.

Impact: faster attention and much lower memory use, which especially helps long contexts. It’s “exact” (not an approximation — no quality loss). Modern engines use FlashAttention (and successors like FlashAttention-2/3, and FlashInfer) under the hood. You rarely invoke it directly; you benefit by using an engine that ships it. (Attacks: attention compute + memory; helps long-context latency.)

5.6 Chunked prefill (stop long prompts from freezing everyone)

Problem it solves: Prefill is compute-heavy (Module 02). A request with a very long prompt does a big prefill that can monopolize the GPU for a moment, stalling the steady decode of all the other requests in the batch — their TPOT spikes, users see stutter. The two phases interfere.

Chunked prefill: break a long prefill into smaller chunks and interleave those chunks with ongoing decode steps, rather than doing the whole prefill in one uninterruptible burst. The big prompt gets processed a slice at a time, leaving room each step for everyone else’s tokens to keep flowing.

Impact: smoother, more predictable TPOT under mixed load; better balance between TTFT of new long requests and decode of existing ones. It’s a key knob for taming tail latency (Module 03). (Attacks: prefill/decode interference; smooths latency.)

5.7 Speculative decoding (more than one token per step)

Problem it solves: Decode is sequential and memory-bound — each step reads all the weights but produces just one token, leaving compute idle (Module 01). What if we could produce several tokens per expensive step?

Speculative decoding: use a small, fast draft model (or a cheap heuristic) to guess the next several tokens quickly. Then the big, expensive target model checks all those guesses in a single parallel pass (verification is parallel, like prefill, so it’s cheap relative to the savings). Every guess the big model agrees with is accepted for free; at the first disagreement, it corrects and continues. Because the big model often agrees on easy/predictable tokens, you frequently get 2–4 tokens for the price of one step.

Crucially, the output is identical to what the target model would have produced alone — it’s a speedup, not an approximation. Variants include using a tiny draft model, using the model’s own earlier layers, n-gram/lookup methods, and Medusa/EAGLE-style extra prediction heads.

Impact: lower TPOT / latency for a single stream — fantastic for interactive, low-concurrency use. (Attacks: decode; trades spare compute for fewer sequential steps.)

Important caveat: speculative decoding spends extra compute (running the draft + verifying) to save sequential steps. That’s a great deal when the GPU has spare compute — i.e. at low batch size / low concurrency. But at high concurrency, continuous batching has already filled the GPU’s compute with other users’ tokens, so there’s no spare compute to spend, and speculation can even hurt. So: speculative decoding shines for latency-critical, lightly-loaded serving; it’s less useful (or counterproductive) for max-throughput, heavily-batched serving. Knowing when not to use it is the expert move.

5.8 Disaggregated prefill and decode

Problem it solves: Prefill (compute-bound, bursty) and decode (memory-bound, steady) have opposite hardware appetites, and on the same GPU they fight — a big prefill stalls decode (the very problem chunked prefill softens). At large scale you can do something more drastic.

Disaggregation: run prefill on one set of GPUs and decode on another, dedicated set. Prefill machines process prompts and produce the KV cache; that cache is transferred to decode machines that stream out tokens. Each pool is tuned for its phase, and the two no longer interfere.

Impact: better, more predictable latency and higher efficiency at large scale; you can also scale the two pools independently to match your workload’s prompt-heavy vs generation-heavy mix. It adds system complexity (you must ship KV caches between machines fast), so it’s a large-scale technique, not a starter one. Newer versions of vLLM and other systems support it. (Attacks: prefill/decode interference at scale.)

5.9 Smaller-but-important techniques

A few more you’ll encounter, briefly:

CUDA graphs: launching thousands of tiny GPU operations each step has CPU overhead. CUDA graphs record a whole step’s sequence of operations once and replay it as a single unit, cutting launch overhead — meaningful for fast decode. (vLLM uses this; it’s why a “graph capture” step appears at startup.)
Kernel fusion: combine several small operations into one kernel so intermediate results stay in fast memory instead of bouncing to VRAM. Less memory traffic, faster.
KV-cache offloading / swapping: when GPU KV memory is full, move cold (inactive) KV cache to CPU RAM or disk and bring it back when needed — trading speed for the ability to handle more/longer sessions. Useful for very long contexts or many idle-but-alive sessions.
Quantization (Module 04): belongs in this toolbox too — smaller weights and KV cache both reduce memory traffic (speed) and free room for bigger batches (throughput).
Multi-LoRA serving: serve many fine-tuned LoRA adapters (small add-on weight patches) on top of one shared base model in the same engine, instead of loading a full separate model per variant. Big memory/cost win when you have many task-specific or customer-specific variants. (vLLM supports this.)

5.10 How they fit together (and the order to reach for them)

These techniques are not alternatives; a good engine stacks them. A sensible mental priority when tuning:

Always-on foundations: continuous batching + PagedAttention + FlashAttention. You get these automatically by using a good engine (vLLM). They’re the baseline, not an upgrade.
Free wins for your workload: turn on prefix caching if you have shared prefixes (system prompts, RAG, chat). Turn on chunked prefill if long prompts are hurting others’ TPOT.
Fit/scale levers: apply quantization (weights and/or KV cache) if memory-bound.
Latency specials: add speculative decoding if you need lower single-stream latency and run at low concurrency.
Large-scale architecture: consider disaggregated prefill/decode and multi-GPU (Module 07) when one box isn’t enough.

The expert framing: you don’t ask “which optimization is best?” You ask “what is my bottleneck right now — TTFT, TPOT, throughput, or fit? — and which technique attacks that?” Module 02’s two questions (prefill vs decode, compute vs memory) plus Module 03’s metrics tell you which bottleneck you’re staring at. The technique follows from the diagnosis.

Check your understanding

Why does static batching waste GPU capacity, and how does continuous batching fix it? (5.2)
What OS idea does PagedAttention borrow, and what two forms of KV-memory waste does it eliminate? (5.3)
For which kind of workload does prefix caching give a big TTFT win — and for which does it do almost nothing? (5.4)
Does FlashAttention change the model’s output? What does it actually optimize? (5.5)
Explain why speculative decoding helps at low concurrency but can hurt at high concurrency. (5.7)
Given a system with great throughput but spiky TPOT whenever long prompts arrive, which one or two techniques would you reach for first? (5.6, 5.8)

Key terms introduced

static vs continuous (in-flight) batching · PagedAttention · KV blocks / paging / block table · copy-on-write KV sharing · prefix caching (APC) · FlashAttention · chunked prefill · speculative decoding / draft model / Medusa / EAGLE · disaggregated prefill/decode · CUDA graphs · kernel fusion · KV offloading/swapping · LoRA / multi-LoRA serving

Next: Module 06 — vLLM Deep Dive, where every idea so far becomes concrete in the world’s most popular open-source serving engine.