Applied AI / AI Production Engineering / Module 01 · Anatomy of an LLM Application
Module 01

Anatomy of an LLM Application

Building Systems AI Production Engineering

Module 01 — Anatomy of an LLM Application

Goal of this module: Open up a real AI application and understand every part, and trace how a single user request flows through the whole thing. You cannot observe, secure, or budget for a system you cannot picture. By the end you will be able to draw the architecture from memory.

1.1 The simplest possible AI app

Let’s build the mental picture from the ground up. The simplest AI app has three parts:

  1. A user who types something.
  2. Your code that takes what they typed, wraps it in some instructions, and sends it to a model.
  3. The model (usually run by a provider like OpenAI, Anthropic, or Google, accessed over the internet) that sends back an answer.

In pseudo-code:

user_message = "How do I reset my password?"

prompt = "You are a helpful support agent. Answer the user's question.\n" + user_message

answer = call_model(prompt)   # this goes over the internet to the provider

show(answer)

That’s it. That is a working AI app. And it is also a Level 0 app (recall the maturity ladder): no logging, no cost tracking, no safety checks, no quality measurement. Everything in this tutorial is about responsibly growing this seed into a real system. To do that, we first need names for all the parts.

1.2 The anatomy, part by part

Here are the components you will find in almost every production LLM application. Not every app has all of them, but knowing them all gives you the full vocabulary.

The model (the LLM itself)

The trained brain. In production you usually don’t run it yourself — you call a provider’s API over the internet. Key things to know:

  • It’s an external dependency. It can be slow, it can fail, and the provider can change it. (See Module 09.)
  • It has a context window — the maximum number of tokens it can read at once, including your instructions, the user’s message, and any extra data. Think of it as the model’s short-term memory or desk space. If you pile on too much, the oldest stuff falls off the desk.
  • It has knobs you can set, the most important being temperature (how random/creative vs. focused/repeatable the output is — low temperature = more consistent, high = more varied) and max tokens (a cap on how long the answer can be).

The prompt

The full text you send the model. In production this is rarely just the user’s question. It is usually assembled from several pieces:

  • System prompt / instructions: the standing orders. “You are a helpful, concise support agent. Never give legal advice. Always respond in the user’s language.” This is where you encode your app’s behavior, tone, and rules.
  • Context: extra information you inject so the model can answer well — relevant documents, the user’s account details, the current date. (More on this under “Retrieval” below.)
  • Conversation history: in a chat, the previous messages so the model remembers what was said.
  • The user’s current message.

Prompt engineering is the craft of writing these instructions well. It is real and important, but in production it is only the beginning. A perfectly written prompt with no observability, no evals, and no cost controls is still a Level 0 system.

Context assembly / orchestration

The code that builds the prompt and manages the flow. This is the heart of “your code.” It decides which documents to fetch, how much history to include, which instructions to use, and what to do with the model’s answer. As apps grow, this layer becomes a small program in its own right. People often use a framework here (LangChain, LlamaIndex, the provider’s own SDK, or hand-written code) to manage it.

Retrieval and the knowledge base (RAG)

Models only know what they were trained on, and they don’t know your private data (your docs, your policies, your product catalog). The standard solution is RAG — Retrieval-Augmented Generation. In plain terms:

  1. You store your documents in a special database.
  2. When a question comes in, you search that database for the few most relevant chunks of text.
  3. You paste those chunks into the prompt as context.
  4. The model answers using them.

It’s like giving a smart assistant an open-book exam: the model is the smart reader, and retrieval hands it the right pages. The special database is usually a vector database (it stores text as lists of numbers called embeddings that capture meaning, so “car” and “automobile” land near each other). You don’t need the math right now — just hold the picture: retrieval finds the right context; generation writes the answer using it.

Tools and function calling

Modern LLMs can do more than talk — they can use tools. You describe some functions to the model (“you can call get_order_status(order_id)” or “you can search_web(query)”), and the model can decide to call them, see the result, and continue. This is how an AI can check a live order status, do math reliably, or take an action in another system. Each tool call is another step in the request that you will want to observe (Module 02–03), cost (Module 05), and secure (Module 06).

Agents

When you let the model loop — think, call a tool, look at the result, think again, call another tool, and keep going until it finishes a task — you have an agent. An agent is an LLM in a loop with tools and a goal. Agents are powerful and increasingly common, but they are also harder to run in production: more steps means more cost, more latency, more places to fail, and more security surface. Much of advanced AI production engineering is really agent production engineering.

Guardrails

Checks that sit around the model to catch problems. An input guardrail might block obviously abusive or malicious messages before they reach the model. An output guardrail might scan the model’s answer for leaked personal data, banned content, or wrong formatting before it reaches the user. Guardrails are a core security and safety tool (Module 06).

The application layer (your actual product)

The user interface, the business logic, the database storing your users and their data, authentication (knowing who the user is), and so on. The AI is usually one feature inside a larger product, not the whole thing.

The observability layer

The logging, tracing, metrics, and dashboards that let you see everything above. This is so central it gets its own modules (02 and 03). For now, just know it wraps around all the other components, recording what happened at each step.

1.3 The life of a request (the most important diagram in your head)

Let’s trace one user message through a realistic production app. Picture each numbered step as a stop on a journey. This sequence is the backbone of everything that follows — observability traces it, cost sums across it, security defends each stop.

USER types a message
        │
        ▼
[1] APPLICATION LAYER receives it
     - Who is this user? (authentication)
     - Are they allowed to do this? (authorization)
     - Are they over their rate limit? (Module 09)
        │
        ▼
[2] INPUT GUARDRAILS
     - Is this abusive, malicious, or an injection attempt? (Module 06)
     - If bad, stop here.
        │
        ▼
[3] CONTEXT ASSEMBLY / ORCHESTRATION
     - Pull conversation history
     - RETRIEVAL: search the knowledge base for relevant docs (RAG)
     - Build the full prompt: system instructions + context + history + message
        │
        ▼
[4] MODEL CALL (over the internet to the provider)
     - Send prompt, get answer
     - This is usually the slowest and most expensive step
     - The model may decide to CALL A TOOL...
        │
        ├──► [4a] TOOL CALL (e.g., look up an order)
        │         - Run the function, get the result
        │         - Send the result back to the model
        │         - (Agents may loop through 4/4a many times)
        ▼
[5] OUTPUT GUARDRAILS
     - Does the answer leak private data? Contain banned content?
     - Is it formatted correctly?
     - Maybe route to a HUMAN for review (Module 07)
        │
        ▼
[6] RESPONSE returned to the user (often STREAMED word by word — Module 08)
        │
        ▼
[7] LOGGING / TRACING / METRICS recorded for the WHOLE journey
     - What was the prompt? The answer? How many tokens? How long? How much $?
     - Capture user feedback (thumbs up/down) if any

Spend real time with this picture. Almost every concept in the rest of the tutorial attaches to one of these steps:

  • Observability & tracing (Modules 02–03) record every step, with timing and detail.
  • Evaluation (Module 04) scores the quality of step 6’s output.
  • Cost (Module 05) sums the tokens and dollars across steps 3, 4, and 4a.
  • Security (Module 06) lives in steps 1, 2, and 5.
  • Human-in-the-loop (Module 07) is an optional branch at step 5.
  • UX (Module 08) is how steps 1 and 6 feel to the user.
  • Reliability (Module 09) is about steps 4 and 4a failing gracefully.

1.4 Where things go wrong (a preview)

To motivate the modules ahead, here is where each part of the journey commonly breaks in production:

  • Step 3 (retrieval): the wrong documents get pulled, so the model answers from bad context. “Garbage in, garbage out.” A huge fraction of RAG quality problems are actually retrieval problems, not model problems.
  • Step 4 (model): the model hallucinates, is slow, returns malformed output, or the provider has an outage.
  • Step 4a (tools): the model calls the wrong tool, with wrong arguments, or a tool fails and the model doesn’t handle it.
  • Step 5 (output): something harmful or private slips through because there is no output guardrail.
  • Across all steps: costs balloon because each step adds tokens, and nobody is watching the total.
  • Everywhere: without step 7 (logging/tracing), you cannot tell which of the above happened. This is why observability is the foundation.

1.5 Build vs. buy: the component landscape

You will rarely build all of this from scratch. The ecosystem (as of this writing) breaks down roughly like this — names will change, categories will not:

  • Model providers: OpenAI, Anthropic, Google, and others; plus open-weight models (Llama, Mistral, Qwen) you can run yourself.
  • Orchestration frameworks: LangChain, LlamaIndex, or the provider SDKs, or your own code.
  • Vector databases (for RAG): Pinecone, Weaviate, Qdrant, pgvector, and many more.
  • Observability / tracing for LLMs: Langfuse, LangSmith, Arize Phoenix, Datadog, Helicone, and others (Modules 02–03).
  • Evaluation tools: often the same platforms as above, plus dedicated eval frameworks (Module 04).
  • Guardrail tools: provider safety filters, NeMo Guardrails, Guardrails AI, and custom checks (Module 06).
  • Gateways / proxies: a single chokepoint all model calls pass through, useful for cost control, caching, routing, and security (Modules 05–06).

Expert habit: Understand the category and what problem it solves, not just one product name. Tools come and go; the role they play in the architecture is durable.

1.6 Try this

  1. Draw it. Without looking, draw the “life of a request” diagram (1.3) from memory. Label every step. Redraw until you can do it cleanly. This diagram is your map for the whole tutorial.
  2. Map a real app. Pick a real AI product. For each component in 1.2, write whether you think the product has it, and why. Which steps in the request journey do you think it runs?
  3. Find the retrieval. Next time an AI assistant cites a specific fact about a company or document, ask yourself: did the model know that, or was it retrieved and pasted into context? How would you tell the difference?

1.7 Self-check

  • Name the components of an LLM app and say in one line what each does.
  • Explain RAG to a non-technical person using the open-book-exam analogy.
  • What is the difference between a tool call and an agent?
  • Walk through the seven steps of a request’s journey out loud.
  • For three of those steps, name a way it can fail and which later module addresses it.

Next: Module 02 — Observability Fundamentals. Now that you can picture the system, we learn how to see it.

← Module 00: Foundations: What AI Production Engineering Really Is