Module 10 — Mastery: Putting It All Together

Goal of this module: Synthesize everything into expertise. We’ll walk a complete, realistic deployment design end-to-end, distill a decision framework you can apply to any new serving problem, give you an expert’s diagnostic checklist for when things are slow or broken, list the common misconceptions that separate beginners from experts, and point you toward continued growth. If you can work through this module comfortably, you’ve arrived.

There’s little new here by design. Mastery isn’t more facts; it’s fluently combining the facts you have to make good decisions under real constraints.

10.1 The unifying mental model (the whole tutorial in one picture)

Everything you’ve learned hangs off a small number of root facts. Here’s the dependency tree of the entire field:

ROOT FACT 1: LLMs generate one token at a time (autoregressive)   [Mod 00]
ROOT FACT 2: GPUs are massively parallel; have separate
             compute and memory budgets                            [Mod 01]
        │
        ▼
KEY CONSEQUENCE: decode is MEMORY-BOUND, prefill is COMPUTE-BOUND,
                 and the KV CACHE is the memory that grows & limits [Mod 02]
        │
        ├──► so we MEASURE with TTFT/TPOT/throughput/goodput,
        │    and trade latency vs throughput                        [Mod 03]
        │
        ├──► so we SHRINK memory traffic & size via QUANTIZATION     [Mod 04]
        │
        ├──► so we OPTIMIZE: continuous batching + PagedAttention
        │    (manage KV memory) + prefix caching + speculative ...   [Mod 05]
        │
        ├──► all of which vLLM IMPLEMENTS and we CONFIGURE           [Mod 06]
        │
        ├──► and when one GPU can't fit/serve it, we DISTRIBUTE
        │    (TP/PP/DP/EP matched to interconnect)                   [Mod 07]
        │
        ├──► and we OPERATE it (containers, K8s, autoscale, observe,
        │    secure, deploy safely)                                  [Mod 08]
        │
        └──► and we MEASURE & COST it to plan capacity and defend
             decisions: goodput per dollar within SLO                [Mod 09]

If someone asks you why any technique exists, you can trace it back up this tree to a root fact. That traceability — not memorized flag lists — is what mastery feels like.

10.2 A complete worked design

Let’s design a real deployment from scratch, using the Module 3.9 framing and pulling in every module. This is the kind of document an expert produces at the start of a project.

The brief

“We’re launching an internal coding assistant for 2,000 engineers. It uses a 70B-class model. Each request has a long shared system prompt (~1,500 tokens of instructions + style guide) plus the user’s code context (~2,000 tokens), and generates ~400 tokens. Peak ~40 requests/sec. Responses must feel snappy: p99 TTFT < 800 ms, p99 TPOT < 40 ms. Budget-conscious but quality matters.”

Step 1 — Write down the four things (Module 3.9)

Workload: input ~3,500 tokens (1,500 shared + 2,000 unique), output ~400 tokens, ~40 req/s peak, online/interactive, heavy shared prefix.
Latency SLO: p99 TTFT < 800 ms, p99 TPOT < 40 ms.
Throughput target: sustain 40 req/s at peak with headroom.
Budget/hardware: cost-conscious; quality matters → can’t over-quantize blindly.

Step 2 — Model fit & precision (Modules 01, 04)

70B in FP16 ≈ 140 GB → won’t fit one 80 GB GPU.
Options: (a) FP8 ≈ 70 GB → fits one 80 GB GPU but leaves little KV room for concurrency at 3,500-token contexts — risky. (b) Tensor-parallel across 2–4 GPUs in FP8/BF16 for comfortable KV headroom and more bandwidth. (c) INT4 ≈ 35 GB → roomy on one GPU, but validate coding quality carefully (4-bit can hurt precise code).
Decision: start with FP8 on TP=2 (two 80 GB GPUs per replica): fits with generous KV room for long contexts and concurrency, near-lossless quality, doubled bandwidth helps TPOT. Keep INT4-single-GPU as a cost-saving candidate to A/B on quality later.

Step 3 — Optimizations to enable (Modules 05, 06)

Continuous batching + PagedAttention + FlashAttention: automatic in vLLM. ✔
Prefix caching: definitely on — the 1,500-token system prompt is shared across every request, so caching it slashes prefill and crushes TTFT. This single feature is the biggest TTFT win available here. ✔ (--enable-prefix-caching)
Chunked prefill: on — 3,500-token prompts are long; chunking keeps them from stalling others’ TPOT, protecting the 40 ms p99 TPOT target. ✔ (--enable-chunked-prefill)
KV-cache FP8: consider, to fit more concurrency at long context. Validate quality. (--kv-cache-dtype fp8)
Speculative decoding: maybe — interactive and possibly moderate concurrency, so it could cut TPOT. But at peak batching there may be no spare compute (Module 5.7). Decision: benchmark with and without; enable only if it helps at realistic concurrency.

Step 4 — vLLM config sketch (Module 06)

vllm serve <70B-model-fp8> \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \              # covers 3,500 in + 400 out + margin
  --enable-prefix-caching \          # shared system prompt → TTFT win
  --enable-chunked-prefill \         # protect TPOT under long prompts
  --kv-cache-dtype fp8 \             # more concurrency at long context (validate)
  --max-num-seqs 48                  # tune via benchmark to the SLO knee

Step 5 — Distribute & scale (Modules 07, 08)

Each replica = TP=2 (one node, NVLink between the 2 GPUs — keep TP intra-node, Module 7.8).
Data-parallel replicas behind a load balancer for throughput (Module 7.6).
Kubernetes: each replica a pod requesting 2 GPUs; readiness probes gate traffic until the 70B model loads; autoscale on queue length / KV-cache utilization / TTFT-vs-SLO (Module 8.4), with warm headroom because 70B load is slow.

Step 6 — Benchmark & size (Module 09)

Benchmark one replica at the real length profile (3,500 in / 400 out), sweeping concurrency to find the knee under p99 TTFT < 800 ms and p99 TPOT < 40 ms. Suppose it sustains ~12 req/s per replica within SLO.
Replicas for 40 req/s peak: 40/12 = 3.3 → 4 replicas × 1.3 safety ≈ 5 replicas = 10 GPUs at peak; autoscale down off-hours.
Cost per token from the knee throughput × GPU hourly rate (Module 9.6); compare against the INT4-single-GPU candidate (5 GPUs instead of 10 — big saving if coding quality holds, so A/B it).

Step 7 — Operate (Module 08)

Containerized image, weights from object storage, /metrics → Prometheus → Grafana dashboards for TTFT/TPOT/goodput/queue/KV%.
Alerts on p99 TTFT approaching SLO and KV utilization > 95%.
Canary new model/quant versions on a slice of traffic, watching quality and latency, with instant rollback.
Auth via gateway + per-engineer rate limits; private network; guard against prompt injection given it reads code context.

That document — fit → precision → optimizations → config → distribution → benchmark/size → operate — is a complete, defensible serving design. Every choice traces to a concept and a measurement. Produce one of these and you’re operating as an expert.

10.3 The decision framework (apply to any serving problem)

Distilled to a repeatable procedure:

State the brief (Module 3.9): workload shape, latency SLO, throughput target, budget/hardware.
Check fit (Modules 01–02): weights + KV cache vs available VRAM at your context length and target concurrency.
Pick precision (Module 04): highest quality that fits and hits your speed/cost goals; validate quality on your task.
Choose parallelism only as forced (Module 07): quantize → TP in a node → DP replicas → PP across nodes, escalating reluctantly.
Enable optimizations matched to your bottleneck (Module 05): prefix caching for shared prompts, chunked prefill for long prompts, speculative decoding for low-concurrency latency, etc.
Configure vLLM accordingly (Module 06), changing one knob at a time.
Benchmark honestly under realistic load (Module 09): find the knee; size capacity with a safety factor.
Operationalize (Module 08): containerize, orchestrate, autoscale on LLM-aware signals, observe, secure, deploy safely.
Compute cost and iterate (Module 09): goodput per dollar within SLO; revisit precision/optimization/hardware to improve it.

Whenever you face a new model, new hardware, or new requirements, run this loop. It never goes stale, even as specific tools and GPUs change.

10.4 Expert diagnostic checklist: “it’s slow / broken”

When a serving system misbehaves, diagnose with these, roughly in order. Each maps to earlier modules.

“TTFT is too high”:

Queue wait? → capacity-limited; scale out or raise concurrency limits (Mod 08).
Long prompts? → enable/verify prefix caching (shared prefixes) and chunked prefill (Mod 05).
Prefill compute-bound on weak GPU? → consider more compute or TP (Mod 01/07).

“TPOT is too high / output streams slowly”:

Memory-bound decode (expected) — is bandwidth the limit? → quantize weights/KV, or add TP for more bandwidth (Mod 04/07).
Batch too large for your latency target? → lower max-num-seqs (trade throughput, Mod 03/06).
Long-prompt prefills stalling decode? → chunked prefill (Mod 05).

“Throughput is low / GPUs underused”:

Benchmarking single-request? → test under concurrency (Mod 09).
Batches too small? → raise gpu-memory-utilization / max-num-seqs (Mod 06).
KV memory the limit? → quantize KV cache, lower max-model-len, PagedAttention already on? (Mod 04/05/06).

“Out of memory”:

KV cache grows with concurrency & context (Mod 02) — lower gpu-memory-utilization, max-model-len, or max-num-seqs; quantize; or add GPUs (Mod 04/06/07).

“Quality regressed after a change”:

Over-aggressive quantization? Validate on your task; back off precision (Mod 04).
New model/version in deploy? Canary caught it? Roll back (Mod 08).

“Won’t start on the GPU”:

Driver/CUDA/library mismatch — the #1 cause (Mod 06/08). Check nvidia-smi, image versions.

Notice every fix is a concept you understand, not a magic incantation. That’s the whole point of building bottom-up.

10.5 Common misconceptions (beginner → expert deltas)

❌ “More GPU compute (TFLOPS) = faster generation.” ✅ Decode is memory-bound; bandwidth usually matters more for TPOT (Mod 01/02).
❌ “Bigger batches always make things slower for users.” ✅ With continuous batching, you gain large throughput for modest latency cost; the trick is operating at the knee (Mod 03/05/09).
❌ “Quantization just makes models worse.” ✅ Done well (AWQ/GPTQ/FP8), quality loss is often negligible and it speeds decode and frees KV memory (Mod 04).
❌ “100% GPU utilization means we’re efficient.” ✅ That number lies for memory-bound decode; measure throughput vs the hardware ceiling (Mod 03/09).
❌ “Speculative decoding always helps.” ✅ Only with spare compute (low concurrency); useless or harmful when already batched full (Mod 05).
❌ “More GPUs = more performance.” ✅ Over-sharding adds communication overhead and diminishing returns; use the least parallelism that meets the need (Mod 07).
❌ “We benchmarked one request, it’s fast, ship it.” ✅ Meaningless without concurrency, length profile, and percentiles (Mod 09).
❌ “A new model is just a config swap.” ✅ It can change output quality; canary and evaluate on real tasks (Mod 04/08).

Internalizing these eight reversals is, honestly, a large fraction of what separates an expert from a confident beginner.

10.6 Where to go next (continuing to mastery)

You now have the durable foundation. To keep growing:

Get hands-on. Stand up vLLM on a real GPU (cloud or local), serve a small model, then reproduce the Module 09 latency-throughput curve yourself. Touching it cements everything.
Read the primary sources. The PagedAttention/vLLM paper, the FlashAttention papers, and the official vLLM docs (https://docs.vllm.ai) deepen the intuition you’ve built. You’ll now read them with understanding instead of intimidation.
Follow the moving edge. The field changes fast: new quantization formats, new attention kernels, disaggregated serving, better MoE serving, new hardware. Your framework (Section 10.3) absorbs new techniques — each new trick just slots onto the dependency tree (Section 10.1). When you meet something new, ask: which root fact does it exploit, which bottleneck does it attack, prefill or decode, compute or memory?
Learn the neighbors. Adjacent skills compound your value: RAG systems, structured/constrained generation, evaluation/quality measurement, fine-tuning and LoRA serving, and the alternative engines (TensorRT-LLM, SGLang, TGI) whose concepts you already share.
Practice the economic framing. Always tie performance to goodput per dollar within SLO. Engineers who connect infrastructure to cost and user experience are the ones who get trusted with the big decisions.

10.7 Final word

You started this tutorial at zero — not knowing what a token was. You can now reason from the physics of GPU memory all the way up to a costed, production, multi-GPU vLLM deployment, and diagnose it when it misbehaves. More importantly, you can reason about techniques that didn’t exist when this was written, because you understand why the field works the way it does, not just what its current tools are called.

That “why,” traced through the dependency tree in Section 10.1, is exactly what an industry expert carries in their head. Keep building, keep measuring, and keep asking which root fact each new idea exploits.

Welcome to the field. Go serve some tokens.

Check your understanding (capstone)

Trace any one optimization technique back up the dependency tree (10.1) to a root fact. (10.1)
Given a fresh serving brief, list the nine steps of the decision framework from memory. (10.3)
A system has great throughput but p99 TPOT spikes whenever long prompts arrive. Diagnose and propose two fixes. (10.4)
Pick three misconceptions from 10.5 and explain the expert correction for each. (10.5)
Reproduce the worked design’s reasoning for why prefix caching is the single biggest TTFT win in that scenario. (10.2)

Key terms introduced

dependency tree of the field · decision framework · diagnostic checklist · goodput per dollar within SLO · (synthesis of all prior modules)

You’ve reached the end of the modules. Return to the index any time, and keep the glossary handy as you practice.