Module 11 — Production Best Practices

Goal: Take prompts from “works in a chat window” to “runs reliably, cheaply, and fast in a real product.” This module covers cost, latency, model selection, caching, versioning, monitoring, and the operational habits of professional prompt engineers.

11.1 The shift from playground to production

In a chat window you care about one good answer. In production you care about thousands of answers per day across diverse inputs, under constraints of money, speed, reliability, and safety. The discipline that makes this work is sometimes called LLMOps (LLM operations) — the prompt-engineering side of shipping AI features. Everything here builds on Modules 9 (evals) and 10 (safety).

11.2 Managing cost

You pay per token (input + output), and costs add up fast at scale. Levers to control cost:

Trim the prompt. Remove redundant instructions, verbose examples, and unnecessary context. Every token in every call costs money. (But don’t trim so far that quality drops — measure with evals.)
Limit output length. Set max_tokens and ask for concise answers; long generations are often the biggest cost.
Right-size the model (see 11.4). Don’t use your most expensive model for easy tasks.
Few-shot economy. Examples help quality but cost tokens on every call. Use the minimum that maintains quality; consider fine-tuning if you need many examples repeatedly.
Caching (see 11.5). Avoid paying to recompute the same thing.
Retrieve, don’t dump. In RAG, send the few relevant chunks, not entire documents (Module 8).
Batch where the provider supports cheaper asynchronous batch processing for non-urgent work.
Cap and monitor spend. Set budgets/alerts; rate-limit to prevent runaway costs and abuse (Module 10).

Track cost per request as a first-class metric in your evals (Module 9). A prompt that’s 3× cheaper for the same quality is a real win.

11.3 Managing latency (speed)

Slow responses hurt user experience. Reduce latency by:

Stream the output. Show tokens as they’re generated so users see progress immediately, even if total time is unchanged. Big perceived-speed win.
Shorten outputs. Generation time scales with output length (each token is produced sequentially). Concise = faster.
Use smaller/faster models for latency-sensitive steps.
Reduce steps. Each call in a chain/agent adds latency. Combine steps where safe; parallelize independent calls (Module 7).
Cache frequent results (11.5).
Set timeouts and fallbacks so a slow/failed call degrades gracefully instead of hanging.

There’s often a cost ↔ latency ↔ quality triangle. Bigger models = better quality but slower and pricier; chains = better quality but more latency/cost. Choose deliberately per use case, guided by evals.

11.4 Choosing the right model

There’s rarely one “best” model — there’s the right model for this task under your constraints. Consider:

Capability needed: Hard reasoning/coding may need a top-tier or dedicated reasoning model; classification/extraction/formatting often run fine on a small, cheap, fast model.
Cost and latency budget: smaller models are cheaper and faster.
Context window needed: large documents need large-context models.
Modalities: need image/audio input? Pick a multimodal model.
Privacy/deployment: can data go to a third-party API, or do you need an open-weight model you host yourself?
Reliability of structured output / tool calling: some models follow schemas and call tools more reliably.

Practical pattern — model routing/cascade: use a cheap model for easy cases and escalate to a stronger model only when needed (e.g., when confidence is low or the router flags complexity, Module 7). This often cuts cost dramatically with little quality loss. Always re-run your evals when you switch or upgrade models — behavior changes between versions.

11.5 Caching

Don’t pay (in time or money) to compute the same thing twice.

Exact-match caching: if the same input comes in again, return the stored output. Great for FAQs and repeated queries.
Semantic caching: if a similar question comes in (by embedding similarity), reuse a prior answer. More hits, but be careful it’s truly equivalent.
Prompt/prefix caching: many providers can cache a long, unchanging prompt prefix (e.g., a big system prompt or fixed context) so you’re not billed full price to reprocess it each call. Structure prompts to keep the stable part first to benefit.

Cache cautions: invalidate when underlying data changes; don’t cache personalized/sensitive responses across users; watch for serving stale info.

11.6 Prompt versioning and management

Treat prompts as versioned assets, like code:

Store prompts in version control (or a prompt-management tool), not pasted ad hoc in code. Track who changed what and why.
Version your prompts (v1, v2…) and tie each to its eval results, so you can compare and roll back.
Separate prompts from code where practical (templates with variables) so non-engineers can iterate and you can change prompts without redeploying everything.
Use templates with clear variable slots (e.g., {user_question}, {context}) and sanitize/validate what you insert (Module 10).
Document each prompt: its purpose, expected inputs/outputs, model and settings it’s tuned for, and known limitations.
Pin model + settings alongside the prompt — a prompt tuned on one model/temperature may behave differently elsewhere.

11.7 Reliability and robustness in production

Real inputs are messy and APIs occasionally fail. Build for it:

Validate outputs (Module 5) and have repair/retry paths.
Handle API errors and rate limits with retries (exponential backoff) and graceful fallbacks (a default message, a cheaper model, or a human handoff).
Set timeouts. Don’t let a hung call freeze the experience.
Design for non-determinism: don’t assume identical output each run; build logic that tolerates variation (or lower temperature for consistency).
Fallback chains: e.g., try primary model → on failure, try secondary → on failure, return a safe default.
Idempotency for actions: ensure a retried action (Module 6) doesn’t, say, send two emails or charge twice.

11.8 Monitoring and observability

You can’t manage what you don’t watch. In production, log and monitor:

Inputs, outputs, and tool calls (with privacy care — redact sensitive data) for debugging and building eval sets from real traffic.
Metrics: latency, cost/tokens per request, error/failure rates, valid-output rate, tool-success rate.
Quality signals: user feedback (thumbs up/down), thumbs-down reasons, escalation/abandonment rates, and periodic automated evals on sampled traffic (Modules 9).
Safety signals: flagged content, blocked outputs, suspected attacks (Module 10).
Drift: watch for quality changes over time (input distribution shifts, provider model updates). Alert on regressions.

Close the loop: mine logs for failures → add them to your eval set → improve the prompt → re-deploy → keep watching. This continuous loop is how production systems actually get good.

11.9 Fine-tuning vs. prompting (when to consider it)

Mostly you’ll solve problems with prompting + RAG + tools. Consider fine-tuning (further training a model on your examples) when:

You need a very consistent style/format and prompting alone is unreliable, or
You have many high-quality examples and want to bake them in (cheaper per call than huge few-shot prompts), or
You need a smaller/cheaper model to match a bigger one’s quality on a narrow task.

Don’t fine-tune just to add knowledge — RAG is usually better, cheaper, and more current for facts (Module 8). Fine-tuning is for behavior/format, retrieval is for knowledge. Fine-tuning adds data-prep, training, and maintenance overhead; reach for it only after prompting/RAG hit their limits, and keep your evals to prove it actually helped.

11.10 A production launch checklist

[ ] Clear success criteria and an eval set (incl. edge + adversarial cases)   (Mod 9, 10)
[ ] Prompt versioned, documented, with model + settings pinned
[ ] Output validation + repair/retry for structured output                    (Mod 5)
[ ] Safety: untrusted-data handling, output moderation, least-privilege tools (Mod 10)
[ ] Cost controls: trimmed prompts, max_tokens, right-sized model, caching, budgets/alerts
[ ] Latency: streaming, concise outputs, timeouts, fallbacks
[ ] Error handling: retries w/ backoff, graceful degradation, idempotent actions
[ ] Monitoring: logs, metrics, user feedback, safety flags, drift alerts
[ ] Human-in-the-loop for high-stakes actions
[ ] Rollback plan and incident process
[ ] Re-run full evals on any prompt or model change

Exercises

Cost audit. Take a prompt you wrote earlier in this course. Cut its token count by 30% without losing quality (trim instructions, shorten examples). Verify quality held using a few test cases.
Model routing. Design a two-tier setup for a support bot: which requests go to a cheap model, which escalate to a strong one, and how you’d decide.
Versioning. Write a short “prompt card” for one of your prompts: purpose, inputs, output format, target model + temperature, known limitations, version.
Failure plan. For a JSON-extraction endpoint, write the pseudocode flow handling: invalid JSON, API timeout, and rate-limit error.
Monitoring plan. List 6 metrics you’d put on a dashboard for a production RAG assistant and what each would warn you about.

Key takeaways

Production prompting optimizes cost, latency, and reliability — not just one good answer.
Control cost with lean prompts, capped outputs, right-sized models, caching, and budgets; improve perceived speed with streaming and shorter outputs.
Pick the right model per task; route cheap→strong to save money. Re-run evals whenever the model or prompt changes.
Treat prompts as versioned, documented assets with pinned model/settings. Validate outputs and handle errors gracefully.
Monitor everything, mine failures back into your evals, and keep high-stakes actions human-gated. Use fine-tuning only when prompting/RAG genuinely fall short.

Next: Module 12 — Capstone Projects & Continued Learning.