Module 01 — Foundations: How LLMs Actually Work

Goal: Understand what a Large Language Model (LLM) is doing under the hood, so your prompts stop being guesswork. You can’t engineer something well if you don’t know how it behaves.

You do NOT need math or coding for this module. Just read carefully.

1.1 What is a Large Language Model?

A Large Language Model (LLM) — like Claude, GPT, Gemini, or Llama — is a computer program trained to do one deceptively simple thing:

Given some text, predict what text most plausibly comes next.

That’s it. At its core, the model is a giant “next-word guesser.” You give it:

The capital of France is

…and it predicts the most likely continuation: Paris.

It learned to do this by reading an enormous amount of text (books, websites, code, conversations) and adjusting billions of internal numbers (“parameters”) until it got really, really good at predicting the next chunk of text.

Why this matters for you: Everything the model does — answering questions, writing code, reasoning — is the side effect of next-token prediction. When you write a prompt, you are not “programming” the model. You are setting up a context so that the most plausible continuation happens to be the answer you want. Good prompting = shaping the context so the right answer becomes the most likely next text.

1.2 Tokens: the model’s unit of text

Models don’t see words or letters. They see tokens. A token is a chunk of text — often a word, part of a word, or punctuation.

Rough rule of thumb in English:

1 token ≈ 4 characters
1 token ≈ ¾ of a word
100 tokens ≈ 75 words

Examples of how text splits into tokens (the · shows token boundaries, approximately):

"hamburger"      ->  ham · burger          (2 tokens)
"unbelievable"   ->  un · believ · able    (3 tokens)
"ChatGPT"        ->  Chat · G · PT          (3 tokens)
"2026"           ->  2026                   (1 token)

Why tokens matter:

You pay per token. APIs charge by input tokens + output tokens. Shorter prompts cost less.
There’s a token limit (see context window below). Both your prompt and the answer must fit.
Weird tokenization causes weird bugs. Models are bad at character-level tasks (counting letters, reversing strings, rhyming) partly because they see tokens, not letters. If you ask “how many r’s in strawberry,” the model isn’t looking at letters — it’s reasoning over tokens. Knowing this tells you when to not trust the model and when to give it a tool instead (Module 06).

1.3 The context window (the model’s “working memory”)

The context window is the maximum amount of text (in tokens) the model can consider at one time. It includes:

[ system prompt ] + [ conversation history ] + [ your new message ] + [ the model's reply ]

All of that must fit inside the context window. Modern models have large windows (tens of thousands to millions of tokens), but they are still finite.

Key consequences:

The model has no memory between separate conversations unless you (or the app) feed prior info back in. Each API call is stateless — the model only “knows” what’s in the current context window.
In a long chat, early messages can fall out of the window or get less attention. Important instructions placed only at the very top of a very long conversation may be “forgotten.”
“Lost in the middle”: research shows models pay most attention to the beginning and end of a long context, and can overlook information buried in the middle. So put critical instructions at the start AND restate them at the end for long prompts.

Practical takeaway: Treat the context window as the model’s entire universe. If a fact isn’t in the prompt (or retrievable via a tool), the model doesn’t reliably know it.

1.4 How the model generates text: one token at a time

The model produces output autoregressively — one token at a time, feeding each new token back in to predict the next.

Prompt: "Roses are red, violets are"
Step 1 -> predicts "blue"     (context becomes "...violets are blue")
Step 2 -> predicts ","        
Step 3 -> predicts " sugar"
... and so on

At each step the model produces a probability distribution over all possible next tokens. For example:

blue   72%
purple 11%
green   4%
...

Then it picks one. How it picks is controlled by sampling settings (next section). This one-token-at-a-time process is why:

Output streams in word by word.
Once it commits to a token, it can’t go back. If it starts down a wrong path (“The answer is 42 because…”), it tends to justify it. This is why techniques that make the model think before answering (Chain-of-Thought, Module 04) work so well — they let the reasoning happen before the commitment to an answer.

1.5 The settings that control output: temperature, top-p, max tokens

When you call a model (via API or some app settings), you can usually tune a few knobs. Understanding these is core prompt-engineering knowledge.

Temperature (the most important one)

Controls randomness/creativity.

Low (0 to 0.3): Nearly always picks the most likely token. Output is focused, deterministic, repeatable. Use for: math, code, extraction, classification, anything factual or where you want consistency.
Medium (0.5 to 0.8): Balanced. Some variety. Good default for general writing.
High (1.0+): Picks less-likely tokens more often. Output is creative, surprising, sometimes incoherent. Use for: brainstorming, poetry, generating many varied options.

Rule of thumb: If you’d be annoyed getting a different answer each time, lower the temperature. If you want variety, raise it.

Top-p (nucleus sampling)

An alternative/companion to temperature. It tells the model: “only consider the smallest set of top tokens whose probabilities add up to p.” top_p = 0.9 means “ignore the unlikely long tail.” Usually you tune temperature OR top-p, not both. Leaving top-p at its default and just using temperature is fine for beginners.

Max tokens (output length cap)

The maximum number of tokens the model is allowed to generate in its reply. If set too low, the answer gets cut off mid-sentence. If you ask for a long essay, raise it. Note this caps output, separate from the context window.

Stop sequences

Strings that tell the model “stop generating when you produce this.” Useful for structured output and chaining (later modules).

Seed (when available)

Some APIs let you set a seed for (more) reproducible output at a given temperature. Handy for testing.

1.6 Why models “hallucinate” (make things up)

A hallucination is when the model states something false or invented with full confidence — a fake citation, a wrong date, a function that doesn’t exist.

Why it happens: remember the model predicts plausible text, not true text. If a fluent, confident-sounding continuation is statistically likely, the model produces it — even if it’s wrong. The model has no built-in “I’m not sure” sensor and no live connection to facts unless you give it one.

Hallucinations get worse when:

You ask about niche, recent, or obscure facts (thin training data).
You ask for specifics it can’t know (exact statistics, URLs, citations, internal data).
The prompt pressures it to answer no matter what.

How prompt engineering reduces hallucination (you’ll learn the details later):

Give it the facts in the prompt (or via retrieval/tools) instead of relying on memory.
Explicitly allow “I don’t know.” e.g., “If you are not certain, say you don’t know rather than guessing.”
Ask for sources / quotes from provided text only.
Lower the temperature for factual tasks.
Use tools (search, calculator, database) for anything the model can’t reliably recall (Module 06).

Mindset shift: Treat the raw model as a brilliant, fast, confident intern with no internet access and a fuzzy memory. You wouldn’t trust such an intern’s recalled statistics — you’d hand them the document. Same with LLMs.

1.7 What “training” gave the model — and what it didn’t

The model learned from data up to a certain point in time (its knowledge cutoff). It does not automatically know:

Today’s date, news, prices, or events after its cutoff.
Your private documents, your company’s data, your codebase.
The result of any calculation it hasn’t actually performed.

It often will answer such questions anyway (confidently and wrongly). Recognizing the boundary between “the model can know this” and “the model cannot know this” is a core expert skill. When something is outside the model’s knowledge, you must supply it (in the prompt, via retrieval, or via tools).

There are also typically two layers of training that affect prompting:

Pretraining: the giant next-token-prediction phase. Gives raw capability.
Fine-tuning / alignment (e.g., RLHF): a later phase that shapes the model to be helpful, follow instructions, and behave safely. This is why modern chat models follow instructions well and refuse harmful requests. It’s also why “instruction-style” prompts work.

1.8 Base models vs. instruction-tuned (chat) models

Base model: raw next-token predictor. If you type “Write a poem about the sea,” it might continue your sentence rather than obey it. You rarely use these directly today.
Instruction-tuned / chat model: fine-tuned to follow instructions and converse. This is what you interact with in ChatGPT, Claude, Gemini, etc. Everything in this course assumes instruction-tuned chat models.

You’ll also hear about reasoning models (models trained to “think” at length before answering, great at hard math/logic) versus standard models. We’ll touch on choosing models in Module 11.

1.9 The mental model to carry through this whole course

Burn this into your head:

The model is a probability machine that continues text. Your prompt is the context that makes the right continuation the most likely one. You shape behavior by shaping context — through instructions, examples, structure, and the information you provide.

Everything else in this course is a specific technique for shaping context well.

1.10 Mini-glossary (Module 01)

LLM: Large Language Model. A next-token-prediction system trained on lots of text.
Token: the chunk of text a model reads/writes (≈ ¾ of a word).
Context window: max tokens the model can consider at once (prompt + history + reply).
Autoregressive: generating output one token at a time, each feeding the next.
Temperature: randomness knob. Low = focused, high = creative.
Top-p: “nucleus sampling” — only consider the most probable tokens summing to p.
Max tokens: cap on output length.
Hallucination: confident false output.
Knowledge cutoff: the date after which the model has no training knowledge.
Base model: raw predictor. Instruction-tuned model: fine-tuned to follow instructions (what you use).
Inference: the act of the model generating a response (running the model).
Prompt: the input text you give the model. Completion / generation: the output.
System prompt: special top-level instructions that set persistent behavior (Module 03).

Exercises

Token intuition. Guess how many tokens are in: “Prompt engineering is surprisingly fun!” Then look it up with any online tokenizer. Were you close?
Temperature experiment. Ask a chatbot (if it exposes temperature, or via API) the same creative prompt — “Give me a name for a coffee shop” — at temperature 0 and temperature 1, three times each. Note how repeatable vs. varied the answers are.
Hallucination hunt. Ask a model for “three peer-reviewed studies, with authors and years, about [very niche topic].” Try to verify them. Did any turn out fake? This teaches you to never trust unverified citations.
Knowledge boundary. Ask the model something it cannot know (e.g., “What did I eat for breakfast?” or a price today). Notice whether it admits the limit or invents an answer. Then rewrite your question to give it the info it needs.

Key takeaways

Models predict plausible next tokens; they don’t “look up” truth.
Tokens, context window, and temperature are the physics of the system — know them.
Hallucinations come from the prediction nature of the model; you reduce them by supplying facts, allowing “I don’t know,” and using tools.
Your job as a prompt engineer is to shape context so the right output is the most likely output.

Next: Module 02 — Anatomy of a Great Prompt.