Module 03 — Model Serving Fundamentals

Goal of this module: Learn the language and goals of LLM serving. What does it mean to “serve” a model? What metrics define good service (latency, throughput, TTFT, TPOT, goodput)? What is the fundamental tension you are always managing? What is an SLO? By the end you can read a serving dashboard and know what “good” looks like.

This module has almost no new internals — it is about measuring and goal-setting. It turns the machinery from Modules 00–02 into engineering targets.

3.1 What “serving” means

Serving a model means running it as a service: a system that sits there, accepts requests over a network (usually an HTTP API), runs inference, and returns results — continuously, for many users, reliably.

This is different from running a model once in a notebook for yourself. Serving implies:

Many concurrent users, arriving unpredictably.
An API contract — a stable way for clients to send prompts and get answers (very often an OpenAI-compatible API, a de-facto standard shape that most tools, including vLLM, support so clients can switch backends easily).
Performance targets that must hold even under load.
Reliability — it should not fall over, leak memory, or hang.

The software that does all this is a serving engine or inference server. vLLM is one. Its job is to take incoming requests and turn them into efficient GPU work, manage memory (especially the KV cache), batch requests together, and stream results back.

3.2 The two things users feel: latency and throughput

There are exactly two top-level performance properties, and they pull against each other.

Latency — “how fast for one user”

Latency is the time an individual request takes. For LLMs we split it into the two pieces you already met in Module 00, now defined precisely:

TTFT — Time To First Token: time from when the request is sent to when the first output token comes back. Includes any queue wait + prefill time. This is the “pause before it starts typing.” Low TTFT = feels responsive.
TPOT — Time Per Output Token (also called inter-token latency, ITL): the average time between subsequent output tokens during decode. This is the “typing speed.” Low TPOT = answer streams in quickly.

From these you can derive end-to-end latency for a request:

total latency ≈ TTFT + (number of output tokens − 1) × TPOT

So a response with a 0.3 s TTFT and 20 ms TPOT generating 200 tokens takes about 0.3 + 199 × 0.02 ≈ 4.3 seconds total, but the user sees the first word in 0.3 s and then watches it stream. This is why TTFT and TPOT matter separately: they shape the experience, not just the total.

A related metric: TPS (tokens per second) for a single stream = roughly 1 / TPOT. If TPOT is 20 ms, that’s ~50 tokens/sec for that user.

Throughput — “how much for everyone”

Throughput is the total work the system does across all users, usually measured in total output tokens per second (summed over all concurrent requests), or requests per second. High throughput = you serve more users per GPU = lower cost per request.

Crucial distinction beginners blur: A single user’s “tokens/sec” (≈ 1/TPOT) is a latency measure. The system’s “tokens/sec across all users” is a throughput measure. The same GPU can have modest per-user speed while delivering enormous total throughput, because it is serving hundreds of users at once via batching. Both numbers are “tokens per second” but they mean opposite things. Always ask: per user, or total?

3.3 The fundamental tension: latency vs throughput

Here is the trade-off you will manage for your entire career, and it falls straight out of Module 02’s batching insight.

To get high throughput, you batch many requests together so the GPU’s loaded weights serve everyone at once. Bigger batches = better throughput.
But bigger batches mean each individual request may wait longer to be scheduled, and each decode step does more work, which can raise latency (TTFT and TPOT) for any one user.

So:

  small batches  ──►  great latency, poor throughput (GPU underused, costly)
  large batches  ──►  great throughput, worse latency (users wait more)

Neither extreme is “right.” The art of serving is choosing the operating point that meets your latency targets while maximizing throughput — and using clever techniques (continuous batching, Module 05) that let you get most of the throughput benefit with little of the latency cost. Much of Modules 05–06 is about bending this trade-off curve in your favor.

Reframe: You are not trying to “make it fast.” You are trying to maximize throughput subject to a latency constraint. That sentence is the job.

3.4 SLOs, SLAs, and percentiles

You cannot manage what you do not bound. Enter service targets.

SLO — Service Level Objective: a target you set internally, e.g. “TTFT under 500 ms and TPOT under 50 ms for 99% of requests.” Your tuning goal is to maximize throughput while staying inside the SLO.
SLA — Service Level Agreement: a contractual promise to customers, often with penalties if missed. SLAs are usually looser than SLOs (you keep headroom).

Why percentiles, not averages

Averages lie. If 99 requests are fast and 1 takes 30 seconds, the average looks fine but one user had a terrible time. So serving is measured in percentiles:

p50 (median): half of requests are faster than this. The “typical” experience.
p95 / p99: 95% / 99% of requests are faster than this. The “tail.” These capture the bad experiences that actually drive user frustration and churn.

A good SLO is stated at a percentile: “p99 TTFT < 1 s.” Tail latency (p99) is often where the real engineering battles are — a system can have a lovely p50 and a horrifying p99 because of queueing, big batches, or noisy neighbors. Watch your tails.

3.5 Goodput: the metric that ties it together

Raw throughput can be misleading: a system might process tons of tokens/sec but if half those requests violated the latency SLO, that work is arguably worthless to the user. So a sharper metric is:

Goodput: the throughput of requests that were served within their latency SLO.

Goodput rewards you only for work that was both plentiful (throughput) and good enough (met latency). When you tune a serving system seriously, you are usually maximizing goodput. It is the single best one-number summary of “are we serving well and efficiently.”

3.6 The metrics dashboard you’ll actually watch

Putting it together, here is the core set of numbers an LLM serving engineer monitors. (Module 08 covers how to collect them; here is what they are.)

Metric	Type	Plain meaning	You want it…
TTFT (p50/p95/p99)	Latency	Time to first token	Low
TPOT / ITL (p50/p95/p99)	Latency	Gap between tokens	Low
End-to-end latency	Latency	Whole request time	Low (within SLO)
Throughput (output tok/s)	Throughput	Total tokens generated per second	High
Requests/sec	Throughput	Completed requests per second	High
Goodput	Combined	Throughput meeting SLO	High
Concurrency / active requests	Load	How many requests in flight	Matched to capacity
Queue length / queue wait	Load	Requests waiting to start	Low
GPU utilization	Efficiency	How busy the GPU is	High, but see caveat below
KV-cache utilization	Efficiency	How full the KV memory is	High but not 100% (need headroom)

A famous trap — “GPU utilization” can lie for LLMs. The standard “GPU utilization” number (e.g. from nvidia-smi) only tells you the GPU was doing something, not that it was doing it efficiently. Because decode is memory-bound, a GPU can show 100% “utilization” while its compute units are mostly idle, waiting on memory. So do not declare victory just because utilization is high. Better signals are achieved throughput vs. the hardware’s theoretical ceiling and KV-cache utilization / batch size. We return to honest measurement in Module 09.

3.7 Where time goes: a worked mental model

Let’s trace where a request’s time is actually spent, so the metrics feel concrete.

Request arrives
   │
   ├── QUEUE WAIT ............ (system busy? you wait here)   ─┐
   │                                                          ├─► counts toward TTFT
   ├── PREFILL .............. (process whole prompt)          ─┘
   │
   ├── DECODE token 1 ....... ┐
   ├── DECODE token 2 ....... │  each gap ≈ TPOT
   ├── DECODE token 3 ....... │  (memory-bound, depends on
   ├──   ...                  │   batch size + KV cache size)
   └── DECODE token N ....... ┘
   │
   └── done → free KV cache

Two design implications you can already see:

Long prompts hurt TTFT (more prefill) — so techniques that speed up or reuse prefill (prefix caching, chunked prefill — Module 05) directly improve TTFT.
TPOT depends on how loaded the system is — bigger concurrent batches can raise TPOT. So your latency under load differs from latency when idle. Always benchmark under realistic concurrency, never just one request at a time (a lesson we hammer in Module 09).

3.8 Online vs offline serving

Two operating modes you should distinguish, because they have different goals:

Online (interactive) serving: real users waiting in real time (chatbots, copilots). Latency-sensitive. TTFT and TPOT SLOs dominate. You sacrifice some throughput to keep responses snappy.
Offline (batch) serving: you have a big pile of prompts to process and no one is waiting interactively (e.g. classify a million documents overnight, generate embeddings, evaluate a dataset). Throughput is everything; latency barely matters. You crank batch sizes to the max and let individual requests take as long as needed.

vLLM supports both: an offline mode (a Python library you call to process lists of prompts at max throughput) and an online mode (a running API server for live traffic). Knowing which mode your problem is in tells you which metric to optimize and how aggressively to batch. Many beginners apply online-style caution to an offline job and leave huge throughput on the table, or vice versa.

3.9 A clean way to state any serving goal

Whenever you start a serving project, write down four things. This habit alone marks you as someone who knows what they’re doing:

Workload shape: typical prompt length, typical output length, requests/sec, peak vs average, online or offline.
Latency SLO: target TTFT and TPOT at a stated percentile (e.g. p99).
Throughput / scale target: how many requests or tokens/sec you must sustain.
Budget / hardware: what GPUs you have or can afford.

Every later decision — quantize or not, which optimizations, how many GPUs, how to scale — is downstream of these four. Modules 04–09 give you the tools; this framing tells you which tool to reach for. We will fill this template out fully in the Module 10 capstone.

Check your understanding

Define TTFT and TPOT, and write the formula for total request latency. (3.2)
Why is one user’s “50 tokens/sec” a latency number while a server’s “5,000 tokens/sec” is a throughput number? (3.2)
State the fundamental serving trade-off in one sentence, and explain why batching causes it. (3.3)
Why do serious teams report p99 instead of average latency? (3.4)
What does goodput add on top of plain throughput? (3.5)
Why can a “100% GPU utilization” reading be misleading for LLM decode? (3.6)
For an overnight job classifying a million documents, which metric do you optimize and why? (3.8)

Key terms introduced

serving / inference server · OpenAI-compatible API · latency · throughput · TTFT · TPOT / ITL · tokens per second (per-stream vs total) · latency–throughput trade-off · SLO · SLA · percentiles (p50/p95/p99) · tail latency · goodput · GPU utilization (and its trap) · KV-cache utilization · online vs offline serving

Next: Module 04 — Quantization, our first major optimization lever: shrinking the model so big models fit on small hardware and run faster.