Module 04 — Evaluation & Quality

Goal of this module: Learn how to measure whether your AI is good — systematically, not by vibes. Because AI is probabilistic, you can’t test it like normal code. This module teaches “evals”: what they are, the types, how to build a test set, LLM-as-judge, offline vs. online evaluation, and how evaluation connects to every other pillar. This is the scorecard the whole tutorial relies on.

4.1 Why this module exists

In Module 00 we said AI fails quietly: it returns fluent, confident answers that may be wrong. In Modules 02–03 we learned to capture and see what happened. But seeing isn’t judging. If 10,000 answers go out today, how many were good? You cannot read them all, and “they looked fine when I spot-checked” is not engineering.

Evaluation (or “evals”) is the discipline of measuring AI output quality systematically. It is the bridge between “I can see what my AI did” (observability) and “I know whether my AI is good, and whether my changes made it better or worse” (everything else). Without evals you are flying blind: you can’t safely change a prompt, swap a model to save cost, or prove your system is improving.

Core problem restated: Normal software has a right answer you assert against (assert add(2,2) == 4). AI usually has many acceptable answers and infinite unacceptable ones, and the same input can vary. So we measure quality statistically across many cases, using graders that tolerate variation.

4.2 The mental model: evals are tests for probabilistic systems

Think of an eval as a test that answers: “Across a set of representative inputs, how often does my system produce an acceptable output, scored by some grader?”

Three ingredients:

A dataset — a collection of test inputs (and often expected outputs or reference answers). This is your “exam.”
A grader / scorer — the thing that decides how good each output is. This is the hard, interesting part for AI.
A metric — how you summarize the scores (e.g., “82% passed,” “average helpfulness 4.1/5,” “0 safety violations”).

You run your system over the dataset, grade the outputs, and summarize. Do this every time you change something, and you can prove whether the change helped.

4.3 The three families of graders

How do you score an AI output? There are three broad approaches, from most objective to most flexible.

1. Code-based / deterministic graders

Plain code checks. Use these whenever the quality criterion can be expressed as a rule:

Is the output valid JSON / does it match the required format/schema?
Does it contain (or not contain) a required string, number, or fact?
Is it within a length limit?
For a classification task, does it exactly match the correct label?
Did the system call the correct tool with the correct arguments?

These are cheap, fast, perfectly repeatable, and you should use them as much as possible. Many “AI quality” problems are actually checkable with simple code (e.g., “the answer must include a valid order number”).

2. Human evaluation

People read outputs and rate them. This is the gold standard for quality and nuance — humans catch tone, subtle wrongness, and helpfulness that code can’t. The catch: it’s slow, expensive, and doesn’t scale to every request. Used for:

Building a high-quality reference set (the “ground truth” others are measured against).
Periodic deep audits.
Judging the things only humans can judge well.
Validating that your automated graders agree with human judgment.

Humans are also the source of the feedback signal (thumbs, edits, escalations) you captured in Modules 02–03. That production feedback is a continuous, cheap form of human eval — treat it as precious.

3. LLM-as-judge

Use another LLM call to grade the output. You give a judge model the input, the output (and maybe a reference answer), and a rubric: “Score this support reply 1–5 for helpfulness and accuracy; explain your reasoning.” This is the breakthrough that makes large-scale nuanced evaluation possible — it scales like code but judges fuzzy qualities like a (rough) human.

LLM-as-judge is powerful but must be used carefully:

It’s also probabilistic. The judge can be wrong or inconsistent. Validate it against human judgments before trusting it, and re-validate periodically.
Give it a clear rubric and examples. Vague instructions produce vague, unreliable scores. Specific criteria and sample gradings dramatically improve consistency.
Prefer specific over global judgments. “Does this answer contain any claim not supported by the provided context?” (good, narrow) beats “Is this a good answer?” (vague). Narrow questions are more reliable.
Watch for biases. Judge models can favor longer answers, their own style, or the first option presented. Design rubrics and tests to counter this.
Pairwise often beats absolute. Asking “is answer A or B better?” is frequently more reliable than asking for an absolute 1–10 score.

Expert pattern: Use code-based graders wherever possible, LLM-as-judge for the fuzzy majority, and human eval to calibrate the LLM judge and audit the edges. The three reinforce each other.

4.4 What to actually measure (quality dimensions)

“Good” is not one thing. Break it into dimensions relevant to your app. Common ones:

Accuracy / correctness: are the facts right?
Faithfulness / groundedness (for RAG): does the answer stick to the retrieved context, or did it make things up? (A hallucination check.)
Relevance: does it actually address the user’s question?
Completeness: does it cover what was needed?
Coherence / fluency: is it well-written and clear?
Tone / style: does it match your brand voice and the required register?
Safety: is it free of harmful, biased, or policy-violating content? (Ties to Module 06.)
Format adherence: does it follow the required structure? (Often code-checkable.)
Refusal correctness: does it refuse when it should, and not refuse when it shouldn’t?

For RAG systems specifically, you also evaluate retrieval quality separately from answer quality — recall the three-way split from Module 03. Did retrieval fetch the right documents (measured by metrics like recall and precision over a labeled set)? You can have a great model and still give bad answers because retrieval failed, so measure them apart.

4.5 Building your test set (the dataset)

Your evals are only as good as your dataset. How to build one:

Start small and real. Twenty to fifty genuinely representative examples beat a thousand artificial ones. Pull them from real (logged) production traffic — this is a direct payoff of Module 02’s “capture everything.”
Cover the distribution. Include the common cases and the tricky edge cases: ambiguous questions, ones requiring refusal, ones with no good answer, adversarial inputs, multi-step tasks.
Mine your failures. Every time production produces a bad answer (caught by feedback or review), add it to the test set. This way you never regress on the same bug twice — a regression test, AI-style.
Label where you can. For each case, record what a good answer looks like, or at least the key facts it must/must not contain. This enables code-based and reference-based grading.
Keep it versioned and growing. Your test set is a living asset that grows with your understanding of the product. Treat it like source code.

The flywheel: Production traffic → observability captures it → bad cases get added to the eval set → evals catch those failures before the next release → system improves → repeat. This loop is the engine of AI quality. Most of “becoming an expert” is running this loop well.

4.6 Offline vs. online evaluation

Two complementary modes — you need both.

Offline evaluation runs your system against your fixed test set, before shipping a change. It answers “did my new prompt/model/retrieval setup do better or worse than the old one, on cases I care about?” This is your safety net against regressions (Module 10 wires this into CI/CD). It’s repeatable and controlled, but limited to the cases you thought to include.

Online evaluation measures quality on live production traffic, after shipping. Because you can’t have reference answers for unseen real inputs, online eval leans on:

Real user feedback (thumbs, edits, escalations, retries) — direct signal.
LLM-as-judge running on a sample of live traffic — scoring faithfulness, relevance, safety continuously.
Implicit signals: did the user accept the answer, copy it, complete their task, or give up?
Guardrail trigger rates and business metrics (resolution rate, conversion, etc.).

Offline tells you “is this change safe to ship?” Online tells you “is it actually working for real people, right now?” Mature teams close the loop: online failures feed back into the offline test set (4.5).

4.7 A/B testing and experimentation (preview of Module 10)

When you want to know if version B is really better than version A in the real world, you run an A/B test: route some users to A, some to B, and compare outcomes (quality scores, user feedback, business metrics, cost, latency). This is how you make changes with evidence instead of hope. We go deeper in Module 10 (deployment & lifecycle); here, just connect it: A/B testing is online evaluation used to compare two versions.

4.8 How evaluation connects to every other pillar

Evaluation is the scorecard, so it touches everything:

Observability (02–03): provides the data evals run on and the feedback they use.
Cost (05): you evaluate quality and cost together — a cheaper model is only a win if quality holds. Evals let you safely pursue cost savings.
Security (06): safety evals and red-teaming (deliberately attacking your own system) are evaluation applied to security.
Human-in-the-loop (07): human reviewers generate eval labels; eval scores decide what needs human review.
UX (08): user feedback is both a UX feature and an eval signal.
Reliability (09): you can evaluate behavior under failure (does the fallback still give acceptable answers?).
Deployment (10): offline evals gate releases; online evals and A/B tests validate them.

Without evals, every other pillar is guesswork. “We made it cheaper” — but is it still good? “We added a guardrail” — did it break normal answers? Only evaluation answers these with numbers.

4.9 Common evaluation mistakes

Testing on the cases you already know pass. Build the test set from failures and edges, not happy paths.
Trusting LLM-as-judge blindly without validating it against humans. Calibrate first.
Vague rubrics producing noisy scores. Be specific; give examples.
One overall “is it good” number that hides which dimension regressed. Score dimensions separately.
Evaluating the answer but not retrieval in RAG systems. Measure them apart.
A test set that never grows. If it doesn’t absorb new production failures, it slowly stops reflecting reality.
Only offline or only online. You need both: offline to ship safely, online to know it works for real.

4.10 Try this

Pick your graders. For your support assistant, list three quality dimensions (4.4) and decide for each whether you’d grade it with code, LLM-as-judge, or humans — and why.
Write a judge rubric. Draft the exact instructions you’d give an LLM-judge to score “faithfulness to the provided context” on a 1–5 scale, including what a 1 and a 5 look like. Make it specific enough that two different judges would mostly agree.
Seed a test set. Invent five test cases for the assistant: two common, one ambiguous, one that should be refused, one adversarial. For each, note what a good answer must or must not contain.
Trace the flywheel. Describe, in your own words, how a single thumbs-down in production becomes a permanent regression test.

4.11 Self-check

Why can’t you test AI the way you test normal code?
Name the three families of graders and when to use each.
What is LLM-as-judge, and three cautions for using it well.
Distinguish offline from online evaluation and say why you need both.
Explain the production-to-evals flywheel.
Why must RAG systems evaluate retrieval separately from the answer?

Next: Module 05 — Cost Management. You can now see and judge your system; next, we make sure you can afford to run it.