Module 2 — The Brain: LLMs & Prompting for Agents 🧩

Goal of this module: Understand what a Large Language Model (LLM) actually does, why that matters for agents, and how to “talk” to it well (prompting). You don’t need any math here — just clear mental models.

2.1 What is an LLM, really?

LLM = Large Language Model. Examples: GPT-4, Claude, Gemini, Llama, Mistral.

At its core, an LLM does one surprisingly simple thing:

It predicts the next chunk of text, over and over.

You give it some text, and it guesses the most likely next word (technically, token). Then it adds that word, and guesses the next one, and so on. That’s it. Everything an LLM does — answering questions, writing code, reasoning — comes from this one ability, scaled up massively.

Analogy: It’s like the world’s most advanced autocomplete. Your phone’s keyboard suggests the next word; an LLM does the same but has read a huge chunk of the internet, so its “autocomplete” can write essays, code, and plans.

Why does that make a good “agent brain”?

Because if you train autocomplete on enough text, predicting the next word requires understanding. To correctly finish “The capital of Japan is ___”, the model must “know” it’s Tokyo. To finish “To fix this bug, the first step is ___”, it must reason about code. This emergent reasoning is what we harness for agents.

2.2 Tokens: how LLMs see text

LLMs don’t read letters or whole words — they read tokens. A token is a piece of text, often part of a word.

“cat” → 1 token
“agents” → might be “agent” + “s” → 2 tokens
A rough rule: 1 token ≈ 4 characters ≈ ¾ of a word. So 100 tokens ≈ 75 words.

Why you should care:

Cost: Most AI APIs charge per token (both what you send and what you get back).
Limits: Models have a maximum number of tokens they can handle at once (the context window — next section).
Speed: More tokens = slower responses.

For an agent that takes many steps, tokens add up fast — every “think” and “observe” step consumes tokens. Efficient agents are designed to not waste them.

2.3 The context window: the agent’s short-term memory

The context window is the maximum amount of text (in tokens) the model can “see” at one time. Think of it as the model’s desk space — everything it’s currently working with has to fit on the desk.

What sits in the context window for an agent?

The system prompt (the agent’s instructions / personality)
The goal / user request
The list of available tools and how to use them
The history of everything that’s happened so far (every think, act, observe)
Any retrieved memories or documents

┌──────────── CONTEXT WINDOW (the desk) ────────────┐
│  System prompt: "You are a helpful research agent" │
│  User goal: "Summarize the latest news on X"       │
│  Tools available: web_search, calculator           │
│  Step 1 think/act/observe...                       │
│  Step 2 think/act/observe...                       │
│  ← model reads ALL of this to decide the next step │
└────────────────────────────────────────────────────┘

The catch: The desk is finite. Modern windows range from a few thousand tokens to hundreds of thousands, but they always run out. When an agent runs for many steps, the history grows and can overflow. Managing this is exactly why memory (Module 6) exists — to store things outside the desk and bring them back only when needed.

Key point: Anything not in the context window, the model simply does not know in that moment. The model has no memory between calls except what you put on the desk.

2.4 The three kinds of messages (roles)

When we talk to an LLM in an app, we usually structure the conversation into roles:

System message: The setup/instructions. Defines who the agent is, its rules, its tools. Written by the developer, not the end user. Example: “You are a careful financial assistant. Never invent numbers. Always use the calculator tool for math.”
User message: What the human asks. Example: “What’s 15% tip on $84?”
Assistant message: What the model replies. This can be a final answer or a request to use a tool.

For agents, there’s often a fourth practical role: 4. Tool message: The result returned by a tool, fed back to the model so it can continue. Example: “calculator result: 12.6”

The whole agent loop is really just these messages stacking up in the context window:

system  → "You are an agent with a calculator."
user    → "What's 15% tip on $84?"
assistant → (decides) "Call calculator('84 * 0.15')"
tool    → "12.6"
assistant → "A 15% tip on $84 is $12.60."

2.5 Prompting: how to talk to the brain

A prompt is the text you give the model. Prompt engineering is the craft of writing prompts that get good results. For agents, the system prompt is especially important — it’s the agent’s job description and rulebook.

Here are the core prompting techniques, each with a plain example.

(a) Be clear and specific

❌ “Tell me about dogs.” ✅ “List 3 dog breeds good for apartments, with one sentence each on why.”

The model can’t read your mind. Spell out the format, length, and focus you want.

(b) Give it a role

“You are an experienced travel agent.” This nudges the model into a helpful “mode” and improves relevance.

(c) Show examples (few-shot prompting)

If you want a specific format, show it:

Convert these to JSON.
Input: John, 25  → Output: {"name": "John", "age": 25}
Input: Mary, 30  → Output: {"name": "Mary", "age": 30}
Input: Sam, 40   → Output:

Giving examples is called few-shot prompting. Giving none is zero-shot. Examples dramatically improve consistency.

(d) Ask it to think step by step (Chain-of-Thought)

Adding “Let’s think step by step” makes the model reason out loud before answering, which improves accuracy on hard problems. We’ll go deep on this in Module 5.

(e) Tell it what NOT to do

“Do not make up facts. If you don’t know, say so.” Guardrails in the prompt prevent common failures.

(f) Specify the output format

“Reply with only valid JSON, no extra text.” This is critical for agents, because the program around the agent needs to parse the output reliably.

2.6 A good agent system prompt (annotated example)

Here’s what a beginner-friendly agent system prompt looks like, with comments:

You are ResearchBot, a careful research assistant.   ← role

Your goal is to answer the user's question accurately. ← purpose

Rules:                                                 ← guardrails
- Always use the web_search tool for current facts.
- Never invent statistics or sources.
- If unsure, say "I'm not certain" instead of guessing.

You have these tools:                                  ← tool list
- web_search(query): returns top web results
- calculator(expression): does math

Work step by step:                                     ← reasoning instruction
1. Think about what you need.
2. Use a tool if needed.
3. Read the result.
4. Repeat until you can answer.

When finished, give a short, clear answer with sources. ← output format

Notice it covers: role, goal, rules, tools, process, and output format. A strong agent prompt almost always hits these six.

2.7 “Temperature”: creativity vs. consistency

Most LLM APIs have a setting called temperature, usually from 0 to ~1 (sometimes up to 2).

Low temperature (e.g., 0–0.3): More focused, predictable, repeatable. Best for agents doing precise tasks (math, code, following steps).
High temperature (e.g., 0.7–1.0): More creative, varied, surprising. Best for brainstorming or creative writing.

For agents, you usually want low temperature because you want reliable, consistent decisions — not creative surprises when it’s deciding whether to delete a file.

2.8 What LLMs are bad at (so you design around it)

Knowing the weaknesses helps you build better agents:

Hallucination: Confidently making things up (fake facts, fake citations, fake function names). Fix: give it tools to look things up, and tell it not to guess.
Math: LLMs are unreliable at arithmetic. Fix: give them a calculator tool.
Fresh info: Their knowledge has a cutoff date. Fix: give them web search.
Long memory: They forget anything not in the context window. Fix: add a memory system (Module 6).
Consistency: The same prompt can give slightly different answers. Fix: low temperature, clear instructions, and validation.

Pattern to notice: Almost every weakness is fixed by giving the agent a tool or a memory. That’s the whole reason agents exist — they patch the LLM’s gaps with external help.

2.9 Check yourself ✅

In one sentence, what does an LLM fundamentally do?
What is a token, roughly?
What lives inside the context window during an agent’s run?
Name two LLM weaknesses and the tool that fixes each.

Answers

It predicts the next chunk of text (token), repeatedly.
A piece of text, roughly ¾ of a word / ~4 characters.
System prompt, the goal, the tool list, and the full history of think/act/observe steps (plus any retrieved memory).
Bad at math → calculator tool; outdated knowledge → web search tool. (Also: hallucination → tools + rules; forgetting → memory system.)

2.10 Summary

An LLM is next-token prediction at scale — sophisticated autocomplete that reasons.
Models read tokens; tokens cost money, take time, and fit in a finite context window (the agent’s desk/short-term memory).
We talk to the model via system / user / assistant / tool messages.
Prompting well = be clear, give a role, show examples, ask for step-by-step thinking, set guardrails, fix the output format.
A strong agent system prompt covers role, goal, rules, tools, process, output.
LLMs have predictable weaknesses; we fix them with tools and memory — which is exactly what the rest of this course is about.

Next up: Module 3 — Planning. How does an agent break a big, vague goal into a sequence of doable steps? 👉 03_Planning.md