Module 03 — Tracing Deep Dive

Goal of this module: Master the single most powerful tool for understanding and debugging AI systems. You will learn what traces and spans are, the vocabulary (OpenTelemetry, GenAI conventions), how to instrument an LLM app step by step, and how to actually read a trace to find problems. By the end you could set up tracing for a real app.

3.1 Why tracing deserves its own module

Recall from Module 01 that an AI request is a journey with many stops: input guardrails, retrieval, a model call, maybe several tool calls in a loop, output guardrails, and the response. When something goes wrong — it was slow, it was expensive, it gave a bad answer — the question is almost always which stop caused it?

A pile of separate log lines can’t easily answer that. A trace can, because it stitches all the steps of one request into a single, timed, hierarchical story. For AI systems — especially agents that loop — tracing is not a nice-to-have. It is the way you debug. This is why we give it a full module of its own.

3.2 The core vocabulary: traces and spans

Two words do most of the work.

A span is a single timed unit of work. “Retrieval took 120 ms.” “The model call took 2.1 s.” A span has a name, a start time, a duration, a status (ok/error), and a bag of attributes (extra key-value details, like which model, how many tokens). One step of the request = one span.
A trace is a tree of spans that together represent one whole request. Spans nest: a parent span (the whole request) contains child spans (each step), which may contain their own children (a single iteration of an agent loop containing a model-call span and a tool-call span).

A picture is worth a lot here. A trace for an agentic support request might look like:

TRACE: handle_support_request           (total: 5.4 s)   [root span]
├─ span: input_guardrail                (0.05 s)  ok
├─ span: retrieval                       (0.20 s)  ok   docs_returned=4
├─ span: agent_loop                      (4.90 s)
│   ├─ span: model_call #1               (2.10 s)  ok   in=1200 tok, out=80 tok, $0.012
│   │     └─ (model decided to call a tool)
│   ├─ span: tool: get_order_status      (0.30 s)  ok   order_id=A123
│   ├─ span: model_call #2               (1.80 s)  ok   in=1500 tok, out=140 tok, $0.016
│   └─ span: tool: none (final answer)
├─ span: output_guardrail                (0.05 s)  ok
└─ span: format_and_stream_response      (0.20 s)  ok

Read that top to bottom and you can see the request’s whole life. You can instantly tell that the two model calls dominate the time and cost, that retrieval was fast, and that one tool was used. This is what “observe the trace, not the log line” (Module 02) means in practice.

Key relationships to lock in:

Every span belongs to exactly one trace (identified by a trace ID).
Each span has a span ID and a parent span ID (which builds the tree).
The trace ID is the correlation ID from Module 02 — it ties everything together. Carry it through every step.

3.3 What goes on a span (attributes)

A span is only as useful as the detail you attach. For AI spans, capture (this extends the Module 02 “capture everything” list down to the span level):

On a model-call span:

model and version, provider.
temperature, max_tokens, other settings.
Input prompt (full, assembled) and output (full).
input_tokens, output_tokens, cached_tokens.
cost in dollars.
latency (the span duration), and time to first token if streaming (Module 08).
finish_reason (did it stop naturally, hit the length cap, or call a tool?).
status (ok / error) and any error message.

On a retrieval span:

The search query used.
Number of documents returned and their relevance scores.
Which documents (IDs) — so you can later judge if retrieval pulled the right pages.

On a tool-call span:

Tool name, the arguments the model chose, the result returned, success/failure.

On a guardrail span:

Which checks ran, what they decided (pass/block), and why.

Why this matters: When a user reports “the bot gave me a wrong order status,” you open that trace and immediately see: was the right order ID passed to the tool? Did retrieval pull the wrong doc? Did the model ignore the tool result? The answer is usually obvious once you can see the span attributes — and impossible to find without them.

3.4 OpenTelemetry: the standard worth knowing

You could invent your own tracing format, but there’s a widely adopted open standard called OpenTelemetry (often abbreviated OTel). It defines how to create traces, spans, and attributes in a vendor-neutral way, so you can send the same data to many different backends and switch tools without re-instrumenting your code.

Why you should care as an AI engineer:

Avoid lock-in. Instrument once with OTel; send to Langfuse, Datadog, Grafana, Phoenix, or whatever you choose, now or later.
It’s becoming the AI standard. There is a growing set of GenAI semantic conventions — agreed-upon names for AI attributes (like gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens). Using standard names means tools understand your data automatically.
It unifies AI and the rest of your system. Your AI spans can sit in the same trace as your database queries and API calls, giving you one picture of the whole request, not an AI-only silo.

You do not need to memorize the spec. You need to know: the standard exists, it’s called OpenTelemetry, it has conventions for GenAI, and using it keeps you flexible.

3.5 How instrumentation actually happens

“Instrumentation” just means adding the code that creates spans. There are three common ways, from least to most effort:

Auto-instrumentation / SDKs. Many LLM-observability tools give you a library that automatically wraps your model-provider calls and creates spans for you. You add a line or two of setup, and model calls start showing up as traces. Easiest starting point. Examples: the tracing SDKs from Langfuse, LangSmith, Phoenix, or OpenLLMetry, which auto-patch popular client libraries.
Decorators / wrappers. You mark your own functions (like retrieve() or assemble_prompt()) so they become spans. A common pattern looks like:

@trace                      # this decorator makes the function a span
def retrieve(query):
    docs = vector_db.search(query, k=4)
    span.set_attribute("docs_returned", len(docs))
    return docs

This is how you get your custom steps (retrieval, guardrails, business logic) into the trace, not just the model calls.

Manual spans. You explicitly start and end spans around any block of code, setting attributes by hand. Most control, most effort. You reach for this for unusual steps the libraries don’t cover.

A realistic setup uses all three: auto-instrumentation for the model and database libraries, decorators for your main functions, and a few manual spans for special cases.

The one rule that makes tracing work: propagate the context. Every span must know its parent so the tree assembles correctly. Frameworks handle this for you most of the time, but it’s the thing that breaks when traces show up “flat” (all spans at the top level) or split into pieces. If your traces look wrong, suspect broken context propagation first — especially across async code, background jobs, or service boundaries.

3.6 Reading a trace to solve real problems

Instrumentation is half the skill; reading traces is the other half. Here’s how an expert uses a trace explorer for common AI problems.

Problem: “It’s slow.” Open a slow trace and look at the span durations. The longest span is your culprit. Usually it’s a model call (then: is the prompt too long? is the output too long? is the provider slow right now? are you making calls in sequence that could be parallel?). Sometimes it’s retrieval, or an agent that looped too many times. The trace tells you exactly where the seconds went — no guessing.

Problem: “It’s expensive.” Sort spans by cost (or tokens). You’ll often find one bloated prompt (maybe you’re stuffing the entire conversation history or too many retrieved docs into context) or an agent making far more model calls than necessary. (Cross-reference Module 05.)

Problem: “It gave a wrong answer.” Walk the trace in order:

Retrieval span: did it pull the right documents? If the right info wasn’t retrieved, the model never had a chance — fix retrieval, not the prompt.
Model-call span: given what it was sent, was the answer reasonable? If the context was good but the answer was wrong, it’s a model/prompt problem.
Tool spans: were the right tools called with the right arguments, and did the model use the results? This three-way split (retrieval vs. prompt/model vs. tools) is the most useful debugging move in all of AI engineering, and tracing is what makes it possible.

Problem: “It failed for one user but I can’t reproduce it.” Find their request by user/trace ID, open the exact trace, and read the actual prompt and outputs. Because you captured the real inputs (Module 02), you can see precisely what happened even though re-running might behave differently. This is why we capture content, not just status.

3.7 Tracing agents specifically

Agents (the LLM-in-a-loop from Module 01) are where tracing earns its keep, because they have the most steps and the most ways to go wrong. When tracing an agent, make sure each iteration of the loop is visible: the model’s reasoning/decision, which tool it picked, the arguments, the tool result, and the next decision. Common agent failures you’ll only catch with good traces:

Looping forever / too long: the agent keeps trying and never finishes. The trace shows 15 model calls where 3 should have done it (and 15× the cost).
Wrong tool, wrong arguments: the trace shows the model called refund_order when it should have called get_order_status.
Ignoring tool results: the tool returned the right data but the model’s next step didn’t use it.
Tool errors swallowed: a tool failed, returned an error, and the agent blundered on as if it succeeded.

Without per-iteration spans, an agent is a black box. With them, it’s a readable story.

3.8 Sessions, users, and linking feedback

A single trace is one request. But a user has a conversation (many requests) and an identity. Good AI tracing also captures:

Session/conversation ID: group all the traces in one conversation so you can see the whole back-and-forth.
User ID: find everything a specific user experienced.
Feedback link: attach the thumbs-up/down (or edit, or escalation) to the specific trace it refers to, so you can jump from “this answer was rated bad” straight to “here’s exactly what happened.” This linkage is what turns raw traces into an improvement engine (and feeds your evals — Module 04).

3.9 Common tracing mistakes

Tracing only model calls, not your own steps. Then retrieval and guardrail problems are invisible. Instrument your functions too.
Not capturing prompt/response content on spans (often “to save space”). You lose the ability to debug quality. Capture content; manage size/privacy with sampling and redaction instead of by dropping it.
Broken context propagation producing flat or fragmented traces. See 3.5.
No trace ID surfaced to users/support. Put a (safe) request ID in your UI or error messages so a user report maps directly to a trace.
Collecting traces nobody reads. Make trace review a weekly habit. Tools are useless if no human looks.

3.10 Try this

Trace your example app. On paper, write out the full span tree for one request through your imaginary support assistant (use the example in 3.2 as a template). Include every step from Module 01’s journey.
Diagnose from a trace. Given this trace — root 9.0 s; retrieval 0.1 s; model_call #1 4.2 s (in=8000 tok); tool 0.2 s; model_call #2 4.3 s (in=8200 tok) — what is most likely wrong, and what would you investigate first? (Hint: look at the input token counts.)
Map the three-way split. For a wrong-answer complaint, write the exact questions you’d ask of the retrieval span, the model span, and the tool spans to localize the fault.

3.11 Self-check

Define span and trace, and explain how spans form a tree.
What is a trace ID and why is it the same thing as a correlation ID?
List the attributes you’d put on a model-call span and on a retrieval span.
What is OpenTelemetry and why does using it help you?
Describe the “three-way split” for debugging a wrong answer using a trace.
Name two agent-specific failures that only good tracing reveals.

Next: Module 04 — Evaluation & Quality. Now that you can see what happened, we learn how to judge whether it was any good.