Module 00 — Introduction to LLMs & Inference

Goal of this module: Understand what a Large Language Model is, how it produces text one piece at a time, what the word “inference” really means, and what happens — step by step — when a user sends a request. Everything else in this tutorial builds on the intuition you form here.

No code and no math is required for this module. Just read it slowly.

0.1 What is a Large Language Model, really?

A Large Language Model (LLM) is a computer program that has read an enormous amount of text — books, websites, code, conversations — and, from all that reading, has learned one surprisingly powerful skill:

Given some text, predict what comes next.

That’s it. That is the entire core skill. An LLM is a very, very good “next-word guesser.”

If you type “The capital of France is”, the model has seen that pattern (and millions like it) so many times that it confidently predicts the next word is “Paris”. If you type “Write me a poem about the sea,” it predicts the words of a poem, one after another, because it has seen countless poems and learned what poem-shaped text looks like.

Everything an LLM appears to “do” — answer questions, write code, translate languages, summarize documents — is this same trick applied over and over: predict the next bit of text, add it to what’s there, then predict the next bit again.

Why “Large”?

“Large” refers to two things:

The amount of text it learned from — often trillions of words.
The size of the model itself — measured in parameters.

A parameter is a single number inside the model that got tuned during training. You can think of parameters as the millions or billions of tiny knobs that, together, encode everything the model “knows.” A small model might have 1 billion parameters; a large one might have 70 billion, 400 billion, or more. We write these as 1B, 70B, 405B, where B = billion.

This number matters enormously for infrastructure, because every parameter is a number that has to be stored in memory and used in calculations. More parameters = more memory needed and more computation per word. Hold onto this; it is the thread that runs through the entire tutorial.

0.2 Tokens: the real units LLMs work with

Here is the first place where “predict the next word” was a slight simplification. LLMs do not actually work with words. They work with tokens.

A token is a chunk of text. It might be a whole word, part of a word, a single character, or a punctuation mark. The model has a fixed list of all possible tokens it knows, called its vocabulary (often 30,000 to 200,000+ entries).

Some examples of how text gets split into tokens (this process is called tokenization):

"cat" → one token: cat
"cats" → maybe two tokens: cat + s
"tokenization" → maybe token + ization
"Hello, world!" → Hello + , + world + !

A rough rule of thumb for English: 1 token ≈ 0.75 words, or about 4 characters. So 1,000 tokens is roughly 750 words.

Why you must care about tokens

Tokens are the currency of LLM infrastructure. Almost everything is measured in tokens:

The amount of text a model can consider at once (its context window) is measured in tokens — e.g. “8K context” means 8,192 tokens.
Speed is measured in tokens per second.
Cloud APIs charge per token.
The work the GPU does is per token.

When you read later that “the model generates 50 tokens per second,” that means it is producing about 37 words per second of output. When you read “this model has a 128K context window,” it can take in about 96,000 words of input at once.

Two kinds of tokens to keep straight:

Input tokens (also called prompt tokens): the text you send in.

Output tokens (also called generated or completion tokens): the text the model produces.

They cost different amounts of work, for reasons that become very important in Module 02.

0.3 How an LLM produces an answer: one token at a time

Let’s watch the model answer a question, in slow motion. Suppose the prompt is:

“The sky is”

The model does not produce the whole answer at once. It works in a loop:

Read the prompt (The sky is) and compute a prediction for the single next token.
The prediction is actually a probability for every token in the vocabulary. Maybe blue gets 60%, clear gets 12%, falling gets 0.3%, and so on for all 100,000+ tokens.
Pick one token from that probability distribution (more on how below). Say it picks blue.
Append blue to the text, so now we have The sky is blue.
Repeat: feed The sky is blue back in, predict the next token (maybe . or and), pick one, append it.
Keep going until the model predicts a special end-of-sequence token (its way of saying “I’m done”), or until a length limit is reached.

This loop is called autoregressive generation — “auto” (self) + “regressive” (feeding its own output back as input). This single fact — the model generates one token at a time, each step depending on all previous tokens — is the root cause of almost every performance challenge in LLM serving. We will return to it constantly.

How does it “pick” a token? (sampling)

Step 3 above — choosing one token from the probabilities — is called sampling, and it is controllable:

Greedy / temperature 0: always pick the single most likely token. Deterministic, predictable, can be repetitive.
Temperature: a knob (typically 0 to 2) that controls randomness. Low temperature = safe, focused, repetitive. High temperature = creative, varied, sometimes incoherent. Temperature works by flattening or sharpening the probability distribution before a token is drawn.
Top-p (nucleus sampling): only consider the smallest set of top tokens whose probabilities add up to p (e.g. 0.9), then sample among those. Cuts off the long tail of unlikely, weird tokens.
Top-k: only consider the k most likely tokens.

You do not need to master these now. The key takeaway: the model outputs probabilities; a separate, configurable step turns those into an actual chosen token. This matters for infrastructure because sampling settings affect both output quality and, occasionally, performance.

0.4 What “inference” means

You now know enough to define the central word of this tutorial.

Inference is the act of using a trained model to produce outputs. It is “running” the model. When you send a prompt and get an answer, you have performed one inference.

Contrast it with training, which is the separate, earlier process of creating the model by tuning all those parameters using huge datasets and huge compute. Training happens once (or occasionally, for updates). Inference happens millions or billions of times — every single time anyone uses the model.

	Training	Inference
Purpose	Create / tune the model	Use the model
How often	Rarely (expensive, one-time-ish)	Constantly (every request)
Who cares most	ML researchers	Infrastructure / serving engineers (you)
Cost over a model’s life	Large upfront	Dominant over time, because it never stops

This entire tutorial is about inference infrastructure. We do not train models here. We take an existing model and make it answer requests fast, cheaply, and reliably. Because inference happens so many times, even tiny efficiency gains translate into enormous savings — which is exactly why this field exists and why companies pay experts well to get it right.

0.5 The two phases of inference: prefill and decode

We just saw that generation is a token-by-token loop. But there is an important asymmetry at the very start that you should meet now (and we will go deep on in Module 02).

When a request arrives, inference happens in two phases:

Phase 1 — Prefill (a.k.a. “prompt processing”): The model reads the entire input prompt at once. Because all the input tokens are already known, the model can process them in parallel — all at the same time. This phase produces the first output token. Prefill is compute-heavy but fast relative to its size, because parallel work suits GPUs well.

Phase 2 — Decode (a.k.a. “generation”): Now the model generates output tokens one at a time, in the loop from Section 0.3. Each new token must wait for the previous one, so this phase is sequential — it cannot be parallelized across the tokens of a single request. Decode is where most of the time goes for long answers.

A simple way to feel the difference:

Prefill is like reading a whole page at a glance to get the gist — one big parallel gulp.
Decode is like writing the reply by hand, one word at a time, where each word depends on the last.

This split — a parallel-friendly prefill and a stubbornly sequential decode — is the defining shape of LLM inference. Almost every optimization in this tutorial is a clever way to deal with one phase or the other.

0.6 The lifecycle of a single request

Let’s tie it together by following one user request through a real serving system at a high level. (Each box gets its own deep dive later; this is the map.)

1. User sends prompt  ─────────►  "Summarize this article: ..."

2. Tokenization        ─────────►  text is split into input tokens

3. Scheduling/queueing ─────────►  the server decides when this
                                    request runs, often grouping it
                                    with others (batching, Module 03/05)

4. Prefill             ─────────►  GPU processes all input tokens in
                                    parallel; first output token appears
                                    (this delay = Time To First Token)

5. Decode loop         ─────────►  GPU generates output tokens one by
                                    one; each token streamed back
                                    (gap between tokens = Time Per
                                     Output Token)

6. Detokenization      ─────────►  tokens converted back to readable text

7. Response            ─────────►  user sees the answer (often streaming
                                    in word by word)

Two timings are highlighted because they are the two most important user-facing metrics in all of LLM serving:

Time To First Token (TTFT): how long from sending the prompt until the first word appears. Dominated by prefill (step 4) and any queue wait (step 3). This is what makes a system feel responsive.
Time Per Output Token (TPOT) (also called inter-token latency): the steady gap between each subsequent word during decode (step 5). This is what makes the answer feel like it is “typing” quickly or slowly.

You have already met, in plain terms, the two numbers that Module 03 will formalize and the rest of the tutorial will spend its time minimizing. If you have ever watched ChatGPT pause, then start “typing” a reply word by word — you watched TTFT (the pause) followed by decode at some TPOT (the typing speed).

0.7 Where the difficulty (and the jobs) come from

Why is there a whole field, whole companies, and whole tools like vLLM devoted to “just running the model”? Three reasons, all of which we now have the vocabulary to name:

Models are huge. Billions of parameters must live in fast memory (GPU memory, covered in Module 01). Big models barely fit, or don’t fit, on a single chip.
Decode is sequential and memory-hungry. The one-token-at-a-time loop wastes the GPU’s parallel power and is bottlenecked by memory speed (Module 02). Serving many users at once without slowing each of them down is genuinely hard.
Demand is bursty and costs are high. GPUs are expensive. Squeezing more requests through the same hardware — without blowing past your latency targets — is worth a fortune, which is exactly what serving engineers do.

Every later module attacks one of these three pressures. Quantization (Module 04) attacks #1. Inference optimization (Module 05) and vLLM (Module 06) attack #2 and #3. Multi-GPU serving (Module 07) attacks #1 again for the very biggest models. Production operations (Module 08) keeps it all alive under real traffic.

Check your understanding

Try to answer these before moving on. If you can’t, re-read the linked section.

In one sentence, what is the single core skill an LLM has? (0.1)
Roughly how many words is 4,000 tokens? (0.2)
Why can the model not just produce a whole paragraph in one shot? (0.3)
What is the difference between training and inference, and which one is this tutorial about? (0.4)
Which phase — prefill or decode — can be done in parallel, and which is stuck going one token at a time? (0.5)
What do TTFT and TPOT each measure, and which one corresponds to the “pause before it starts typing”? (0.6)

Key terms introduced

LLM · parameter · token · vocabulary · tokenization · context window · input/output tokens · autoregressive generation · sampling · temperature · top-p · top-k · inference · training · prefill · decode · TTFT · TPOT

All are defined in the glossary.

Next: Module 01 — GPU Fundamentals for AI, where we go down to the hardware and learn what is actually doing all this work.