Module 01 — GPU Fundamentals for AI

Goal of this module: Understand what a GPU is, why LLMs run on GPUs instead of regular processors, the difference between compute and memory, what VRAM and memory bandwidth are, and how to read a GPU spec sheet well enough to predict whether a model will fit and run fast. This is the hardware foundation for everything above it.

You do not need an electronics background. We build from everyday analogies.

1.1 CPU vs GPU: a kitchen analogy

Your computer has a CPU (Central Processing Unit). A CPU is like a small team of brilliant master chefs. Each one is extremely fast and clever and can handle complicated, varied tasks — but there are only a handful of them (a typical CPU has 4 to 64 “cores”).

A GPU (Graphics Processing Unit) is like a stadium full of line cooks. Each cook is individually simpler and less clever than a master chef, but there are thousands of them, and they can all chop vegetables at the same time. A modern GPU has thousands of small cores.

If your task is “cook one incredibly intricate dish that requires 50 sequential steps,” the master chefs (CPU) win.
If your task is “chop 100,000 onions, all identical, all independent,” the stadium of line cooks (GPU) wins overwhelmingly, because they all chop in parallel.

Why LLMs love GPUs

The core operation inside an LLM is matrix multiplication — multiplying big grids of numbers together. A single matrix multiplication is millions of small multiply-and-add operations that are all independent and can happen at the same time. That is exactly the “chop 100,000 identical onions in parallel” situation. So GPUs, originally built to compute the color of millions of pixels in parallel for video games, turned out to be perfect for AI.

Key idea: LLM math is massively parallel. GPUs are massively parallel hardware. That match is the entire reason modern AI runs on GPUs.

1.2 The two things a GPU has that you must track: compute and memory

When you evaluate a GPU for LLM serving, you are really tracking two separate budgets. Confusing them is the #1 beginner mistake, so we separate them clearly.

Budget 1 — Compute (how fast it calculates)

This is the GPU’s raw number-crunching speed: how many math operations per second it can do. It is measured in FLOPS (Floating-point Operations Per Second), usually in the trillions, written TFLOPS (tera = trillion). A GPU might do hundreds or thousands of TFLOPS.

Compute is “how fast are the line cooks chopping.”

Budget 2 — Memory (how much data it can hold, and how fast it can move it)

The GPU has its own dedicated memory, called VRAM (Video RAM) or just GPU memory. This is separate from your computer’s normal RAM. It is measured in gigabytes (GB) — e.g. 24 GB, 80 GB, 192 GB.

Memory has two sub-properties, both critical:

Capacity (GB): how much data can be stored. This decides whether a model fits at all.
Bandwidth (GB/s): how fast data can be moved between the memory and the cores. Measured in gigabytes per second, often in the thousands (e.g. 3,350 GB/s). This decides how fast you can run in many situations.

Memory capacity is “how big is the pantry.” Memory bandwidth is “how fast can ingredients be carried from the pantry to the cooks.”

You will spend your career balancing these. A GPU can be “fast” in compute but “starved” because it cannot move data to the cores quickly enough — the cooks stand idle waiting for onions to arrive. This is so important it has a name, which we meet next.

1.3 The most important concept in the module: compute-bound vs memory-bound

Imagine our stadium of line cooks again. There are two completely different ways the kitchen can be the bottleneck:

Compute-bound: The cooks are all busy chopping non-stop. The pantry can supply onions faster than they can chop. To go faster, you need more or faster cooks (more compute). The cooks are the limit.
Memory-bound: The cooks are fast, but the hallway from the pantry is narrow. Onions trickle in. The cooks spend most of their time standing around waiting for ingredients. Adding more cooks does nothing — they would just stand around too. To go faster, you need a wider hallway (more memory bandwidth). The data delivery is the limit.

Every operation on a GPU is one or the other. And here is the punchline that drives this whole tutorial:

LLM decode (the one-token-at-a-time generation loop) is almost always MEMORY-BOUND.

Why? During decode, to produce just one token, the GPU must read the model’s entire set of weights from memory (all those billions of parameters) but only does a small amount of math with them (because it’s processing a single token). So it reads a huge amount of data and computes relatively little. The cooks (compute) sit idle while the hallway (bandwidth) is jammed moving billions of weights.

This single fact explains an astonishing number of things:

Why memory bandwidth is often the spec that determines decode speed.
Why batching many requests together is so powerful (Module 05) — it lets the GPU reuse weights it already loaded, getting more compute done per byte moved.
Why quantization (Module 04) speeds things up — smaller numbers mean fewer bytes to move through that narrow hallway.

Prefill, by contrast, processes many tokens at once and tends to be compute-bound. Keep this prefill/decode asymmetry from Module 00 in mind — now you can see why they behave differently at the hardware level.

Memorize this: Decode is memory-bound; prefill is compute-bound. If you internalize nothing else from this module, internalize that.

1.4 Where do the model’s weights live? The memory budget

Let’s make the “does it fit” question concrete, because it is the very first thing you check before serving any model.

Every parameter is a number, and a number takes up space depending on its precision — how many bits are used to store it. (We cover precision deeply in Module 04; here is the minimum you need.)

FP32 (32-bit): 4 bytes per parameter. Full precision. Rarely used for serving.
FP16 / BF16 (16-bit): 2 bytes per parameter. The common “default” for serving.
INT8 / FP8 (8-bit): 1 byte per parameter. Quantized.
INT4 (4-bit): 0.5 bytes per parameter. Heavily quantized.

So a quick formula for just the weights:

Weight memory ≈ (number of parameters) × (bytes per parameter)

Worked examples (weights only, in FP16 = 2 bytes each):

Model size	FP16 (2 B)	INT8 (1 B)	INT4 (0.5 B)
7B	~14 GB	~7 GB	~3.5 GB
13B	~26 GB	~13 GB	~6.5 GB
70B	~140 GB	~70 GB	~35 GB
405B	~810 GB	~405 GB	~203 GB

Look at the 70B row. In FP16 it needs ~140 GB just for weights — more than fits on a single 80 GB GPU. So you either quantize it (Module 04) to shrink it, or split it across multiple GPUs (Module 07), or both. You can already see two entire later modules being motivated by one table. That is the point of learning the hardware first.

Important: weights are not the only thing in GPU memory. You also need room for the KV cache (Module 02), activations, and framework overhead. A good rule of thumb: leave meaningful headroom; do not plan to fill 100% of VRAM with weights. We make this precise in Module 09.

1.5 The memory hierarchy inside a GPU (brief but useful)

Memory is not one single thing; it is a hierarchy, fast-and-tiny at the top, slow-and-huge at the bottom. You do not manage this directly, but knowing it exists explains why some optimizations (like FlashAttention, Module 05) work.

        ┌─────────────────────────┐   fastest, tiniest
        │      Registers          │   (per-core scratch)
        ├─────────────────────────┤
        │   Shared memory / L1     │   (per-core-group, very fast, KBs)
        ├─────────────────────────┤
        │       L2 cache           │   (shared, fast, MBs)
        ├─────────────────────────┤
        │   VRAM / HBM (the GB)    │   slowest of the on-card options,
        │                          │   but this is "the memory" people
        │                          │   mean — tens of GB, ~TB/s bandwidth
        └─────────────────────────┘   biggest, slowest
                    │
                    ▼
        ┌─────────────────────────┐
        │ System RAM / CPU / disk  │   off the GPU entirely — very slow
        └─────────────────────────┘   to reach; avoid for hot data

The big GB number on a spec sheet refers to that VRAM/HBM tier. HBM (High Bandwidth Memory) is the special, very fast memory stacked next to data-center GPUs; it is what gives them their enormous bandwidth. A key performance idea: moving data between these tiers is expensive, so good kernels keep data in the fast tiers as long as possible and minimize trips down to VRAM. That is literally what FlashAttention does, and now you have the vocabulary to understand it later.

1.6 CUDA, kernels, and the software that drives the GPU (light touch)

You will hear these words constantly; here is just enough to be fluent.

CUDA: NVIDIA’s software platform and programming model for running general computation on their GPUs. When people say “it requires CUDA,” they mean it runs on NVIDIA GPUs using this stack. (AMD has an equivalent called ROCm.)
Kernel: a small program that runs on the GPU to do one specific operation (e.g. “do this matrix multiply” or “apply this attention”). When someone says “a custom CUDA kernel made it 2× faster,” they wrote a highly optimized GPU program for a hot operation. vLLM ships many such kernels.
Driver & toolkit: the low-level software that lets your operating system talk to the GPU. Version mismatches here are a classic source of “it won’t start” pain in production (Module 08).
Tensor Cores: special units inside modern NVIDIA GPUs that do matrix multiplication on low-precision numbers (like FP16/INT8/FP8) extremely fast. Much of a GPU’s headline TFLOPS comes from Tensor Cores, and using lower precision (quantization) is partly about feeding these units the formats they’re fastest at.

You will rarely write kernels yourself. But you will constantly use tools (like vLLM) whose entire value is that they ship excellent kernels so you don’t have to.

1.7 How to read a GPU spec sheet (the practical skill)

Let’s turn all this into a checklist. When you look at any GPU (data-center or consumer), find these five numbers and you can reason about it:

Spec	What it tells you	Why you care
Memory capacity (GB)	How big a model fits	Decides whether you can serve a model at all
Memory bandwidth (GB/s or TB/s)	How fast weights move	Largely decides decode speed (memory-bound!)
Compute (TFLOPS), at the relevant precision	Raw math speed	Largely decides prefill speed and batch throughput
Supported precisions (FP16, BF16, FP8, INT8, INT4)	Which quantization formats it accelerates	Decides which Module 04 tricks actually speed you up
Interconnect (NVLink / PCIe, and speed)	How fast multiple GPUs talk to each other	Critical for multi-GPU serving (Module 07)

A note on real GPUs (without quoting prices, which change)

You will encounter names from NVIDIA’s data-center line — for example the A100 and H100 (and newer generations), each commonly available in 40 GB or 80 GB (and larger) memory configurations — and consumer cards like the RTX series with smaller memory (e.g. 8–24 GB). The exact current models, memory sizes, and prices change frequently, so always check the vendor’s current spec sheet rather than trusting a number you memorized. The skill that does not go stale is the one you just learned: find capacity, bandwidth, compute, precisions, and interconnect, then reason about fit and speed. If you want current models or prices, look them up at decision time.

Worked reasoning example

“I want to serve a 13B model in FP16. I have one GPU with 24 GB of VRAM and ~900 GB/s bandwidth. Will it work, and roughly how will it feel?”

Fit: weights ≈ 13B × 2 bytes ≈ 26 GB. That is more than 24 GB. ❌ It does not fit in FP16. Options: quantize to INT8 (≈13 GB, now fits with room for KV cache) or use a bigger/second GPU.
Speed feel: decode is memory-bound, so with ~900 GB/s you can estimate a ceiling: roughly, to make one token you read ~13 GB of (INT8) weights, so a rough upper bound is 900 ÷ 13 ≈ ~69 tokens/sec for a single request before overheads. That is a back-of-envelope ceiling, not a guarantee, but it shows how bandwidth → decode speed reasoning works. (We make this estimation rigorous in Module 09.)

That little paragraph is exactly the kind of reasoning an infrastructure engineer does in their head every day. You can now do it.

1.8 Cloud vs on-premise (a quick orientation)

You will get GPUs in one of two ways:

On-premise / owned: you (or your company) buy physical GPUs and run them. Big upfront cost, cheapest per-hour if heavily used, full control. Common at large scale.
Cloud: you rent GPU instances by the hour/second from a provider. No upfront cost, instant scaling, pay for what you use, more expensive per hour. Great for starting out, bursty traffic, and experiments.

For learning, cloud (or a cheap/free GPU notebook) is usually the fastest path to hands-on practice. For production economics, the choice is a real engineering/finance decision we revisit in Module 09 (cost) and Module 08 (operations). You do not need to decide now.

Check your understanding

In the kitchen analogy, what does a GPU correspond to, and why does that suit LLM math? (1.1)
Name the two completely separate “budgets” you track on a GPU. (1.2)
What does it mean for an operation to be memory-bound vs compute-bound? (1.3)
Is LLM decode typically memory-bound or compute-bound? What about prefill? (1.3)
A 70B model in FP16 needs roughly how much memory just for weights? What if you quantize to INT4? (1.4)
Which single spec most directly limits decode speed, and why? (1.7)

Key terms introduced

CPU · GPU · core · parallelism · matrix multiplication · FLOPS / TFLOPS · VRAM / GPU memory · memory capacity · memory bandwidth · compute-bound · memory-bound · precision (FP32/FP16/BF16/FP8/INT8/INT4) · HBM · memory hierarchy · CUDA · ROCm · kernel · Tensor Cores · NVLink · PCIe

Next: Module 02 — Inside the Transformer, where we open up the model itself and meet the KV cache — the single most important data structure in LLM serving.