Module 10 — Deployment & Lifecycle

Goal of this module: Learn to change AI systems safely over time. Prompts, models, knowledge bases, and code all evolve — and in AI, a tiny change can shift behavior in surprising ways. You’ll learn prompt versioning, testing AI changes in CI/CD, safe rollout strategies (canary, A/B, feature flags), monitoring for drift, and how this whole lifecycle ties the other pillars into a continuous improvement loop. This is “MLOps/LLMOps” made practical.

10.1 Why changing AI is uniquely risky

In normal software, you change code, your tests pass, you ship. In AI, three things make change riskier:

Small changes have outsized, unpredictable effects. Reword one line of a prompt and behavior across thousands of cases can shift — better in some, worse in others. There’s no compiler to catch it. You must measure (evals, Module 04).
The model can change underneath you. Providers update models. The exact same prompt and code can behave differently next month with no change on your side. Your system silently drifts. (Module 00’s “the model is a dependency you don’t control,” made concrete.)
The world changes. Your knowledge base goes stale, user behavior shifts, new kinds of inputs appear. A system that was accurate at launch decays over time — this is drift.

The core lifecycle principle: An AI system is never “done.” It must be continuously tested, monitored, and improved. Treat every change — to prompts, models, retrieval, or data — as a release that needs evaluation and safe rollout. Shipping is the beginning of the work, not the end.

10.2 Everything that changes is a versioned artifact

In normal software, you version code. In AI you must version more, because more things affect behavior. Track and version all of these as deliberately as you version code:

Prompts (system instructions, templates) — the most frequently changed and highest-impact artifact. Version them, review changes, and be able to roll back. Don’t leave prompts as untracked strings buried in code that anyone edits silently.
Models — which model and version you use. Pin specific versions where the provider allows, so you’re not silently upgraded. Know exactly what’s in production.
Retrieval configuration and the knowledge base — what’s indexed, how chunks are split, retrieval settings. A change here changes answers.
Tool definitions — the tools/functions the model can call (Module 01).
Guardrail rules (Module 06) and eval sets (Module 04).
The application code itself.

Why this matters: When behavior changes, you need to answer “what changed?” If prompts, models, and retrieval config are all versioned, you can pinpoint the cause and roll back. If they’re scattered and untracked, every behavior change is a mystery. Make your AI system’s behavior reproducible: pin versions, track changes.

10.3 Testing AI changes (evals in CI/CD)

This is where evaluation (Module 04) becomes operational. In normal software, CI/CD (Continuous Integration / Continuous Deployment) runs automated tests on every change and blocks anything that breaks them. For AI, you wire your evals into that pipeline:

On every prompt/model/retrieval change, run your offline eval set (Module 04) automatically.
Compare against the current production version. Did quality go up, down, or sideways? Did any specific case regress?
Block or flag regressions before they reach users. This is your safety net against the “small change, big effect” problem.
Check cost and latency too, not just quality — a change that improves answers but doubles cost or latency may not be worth it (Module 05’s triangle). Evaluate all three together.

Because AI is probabilistic, these “tests” are statistical (Module 04): “did the pass rate / average score hold or improve across the test set?” rather than a single deterministic assertion. Set thresholds and tolerances accordingly. The discipline: no prompt or model change ships without passing evals, the same way no code ships without passing tests.

10.4 Safe rollout strategies

Even after evals pass, don’t flip a change to 100% of users at once — offline evals can’t catch everything (recall Module 04: you need online validation too). Roll out gradually so a bad change hurts few people and can be pulled back fast.

Feature flags: wrap the change so you can turn it on/off instantly without redeploying. The foundation of every safe rollout — if something goes wrong, flip it off in seconds.
Canary release: send the change to a small percentage of traffic first (say 1–5%). Watch your metrics (quality, cost, latency, errors, feedback — all your observability from Modules 02–05). If healthy, expand gradually; if not, roll back having affected almost no one.
A/B test (online experiment): run old (A) and new (B) side by side on comparable user groups and measure which is actually better on real traffic — quality scores, user feedback, business outcomes, cost, latency. This is online evaluation (Module 04) used to make a release decision with evidence instead of hope. The gold standard for “is B really better than A?”
Shadow mode: run the new version alongside production without showing its output to users — just log and evaluate what it would have done. Great for de-risking big changes (a new model, a new retrieval system) with zero user impact before you trust it.
Always keep rollback easy. The single most important safety property: when (not if) a change goes wrong, you can revert quickly. If rollback is hard, every deploy is scary and slow.

The progression: offline evals (Module 04) → shadow/canary on a sliver of traffic → A/B test to confirm → gradual ramp to 100% → keep watching. Each stage catches what the previous couldn’t, at increasing exposure. Never skip straight to 100%.

10.5 Monitoring in production and detecting drift

Shipping isn’t the end (10.1). After a change is live, keep watching — and watch for slow decay even when nothing changed on your side.

Watch your dashboards (Modules 02–05): quality scores, cost, latency, error rates, feedback, guardrail triggers. A regression often shows up here first.
Online evaluation (Module 04): LLM-as-judge and user feedback on live traffic, continuously.
Detect drift:
- Model drift — the provider changed the model and behavior shifted. Catch it by running your evals on a schedule, not only on your own changes. (A sudden score change with no deploy on your side points here.)
- Data drift — inputs are changing (new topics, new phrasing, new languages) and your system handles them worse. Catch it by monitoring the distribution of inputs and where quality drops.
- Knowledge drift — your knowledge base is stale; answers are outdated. Catch it with freshness checks and user feedback.
Alert on regressions so you find out from monitoring, not from angry users.

A signature AI ops scenario: quality drops one morning, but you deployed nothing. The cause is almost always the model changing underneath you or the input distribution shifting. The teams that catch this fast are the ones running scheduled evals (Module 04) on production, not just at deploy time. This is why “the model is a dependency you don’t control” earns constant vigilance.

10.6 The continuous improvement loop (the whole tutorial, in motion)

Lifecycle is where every pillar comes together into one turning loop. This is the engine of an expert AI system:

        ┌─────────────────────────────────────────────┐
        ▼                                             │
  1. OBSERVE production (Modules 02–03)               │
     capture every request, trace, cost, feedback     │
        │                                             │
        ▼                                             │
  2. EVALUATE & find problems (Module 04)             │
     score quality; mine failures, bad feedback,      │
     security findings (06), cost spikes (05)          │
        │                                             │
        ▼                                             │
  3. IMPROVE                                          │
     fix prompts, retrieval, models, guardrails;      │
     add failures to the eval set                     │
        │                                             │
        ▼                                             │
  4. TEST the change offline (evals in CI/CD)         │
        │                                             │
        ▼                                             │
  5. ROLL OUT safely (canary / A/B / flags)           │
        │                                             │
        └─────────────────────────────────────────────┘
              (and back to observing...)

Every pillar plugs in: observability feeds it, evaluation scores it, cost and security and reliability are constraints checked at each step, human-in-the-loop generates labels and approvals, UX captures feedback. This loop, turning continuously, is what separates a Level 4 “optimized” system (Module 00) from a Level 0 “vibes” demo. Mastery is running this loop well, week after week.

10.7 A word on fine-tuning (lifecycle for the model itself)

Most production AI uses models as-is (via prompting and RAG). But sometimes you fine-tune — further train a model on your own examples to specialize it (better at your domain, your format, your tone, or cheaper because a small fine-tuned model replaces a big general one — Module 05). Fine-tuning adds its own lifecycle: you version training datasets, version model checkpoints, evaluate each candidate against your eval set, and roll out the new model with the same safe strategies (10.4). It’s powerful but adds operational weight — reach for it when prompting and RAG have hit their limits, not before. The lifecycle discipline is the same; there’s just one more versioned artifact (the fine-tuned model and its training data).

10.8 Common lifecycle mistakes

Editing prompts directly in production with no versioning, review, or rollback. The single most common and damaging lifecycle sin.
Shipping prompt/model changes without running evals. Flying blind into the “small change, big effect” trap.
Deploying to 100% at once. No canary, no flag, no easy rollback — so a bad change hits everyone.
Not pinning model versions, getting silently upgraded and surprised by behavior changes.
Only evaluating at deploy time, never on a schedule, so model/data drift goes undetected for weeks.
Treating launch as “done”, then watching quality quietly decay.
Not feeding production failures back into the eval set, so the same bugs keep recurring (a broken flywheel).

10.9 Try this

Inventory your artifacts. For your support assistant, list every versioned artifact (10.2) and where/how you’d store and review changes to each.
Design the pipeline. Write the steps that should happen automatically when someone proposes a prompt change, from “they open a change” to “it’s at 100% of users” (combine 10.3 and 10.4). Where can it get blocked, and how do you roll back?
Catch the drift. Describe the monitoring you’d set up to detect, within a day, that your provider silently changed the model and quality dropped. What fires the alert?

10.10 Self-check

Give three reasons changing an AI system is riskier than changing normal software.
List the artifacts you must version beyond code, and why.
Explain how evals fit into CI/CD and what “passing” means for a probabilistic system.
Describe feature flags, canary, A/B test, and shadow mode, and the order you’d use them.
What are model drift, data drift, and knowledge drift, and how do you catch each?
Walk through the continuous improvement loop and name which pillar powers each step.

Next: Module 11 — Capstone & Maturity Model, where we assemble everything into one reference architecture and a roadmap to mastery.