Applied AI / Fine-Tuning & Model Customization / Module 14 · Deployment & Serving
Module 14

Deployment & Serving

Training & Customization Fine-Tuning & Model Customization

Module 14 — Deployment & Serving

Goal: Take your fine-tuned model from a folder on disk to something that actually answers real requests — efficiently, cheaply, and reliably. A model that isn’t deployed creates no value. This module covers packaging (merging, quantizing), serving engines, the powerful multi-LoRA pattern, and the operational realities (cost, latency, monitoring) that distinguish a demo from a product.

1. From trained model to deployable artifact

After Modules 13/07/08 you have one of two things: a base model + a LoRA adapter, or a merged model. Your first deployment decision is whether to merge or keep the adapter separate (Module 07 §4, §7):

  • Merge (merge_and_unload) → a single standalone model, same shape as the base, zero inference overhead. Best when you deploy one specialized model. (Remember the Module 08 §7 recipe: reload the base in 16-bit, attach adapter, merge, then quantize — don’t merge into the 4-bit base.)
  • Keep separate → load the base once and apply the small adapter at request time. Best when you want to serve many specialized models from one base (the multi-LoRA pattern, §5).

Then, almost always, you quantize for serving to cut memory and cost.

2. Quantization for serving (recap + the serving-specific part)

You met this in Module 08 §6. To recap the distinction that matters here: QLoRA quantizes to train; GPTQ/AWQ/GGUF quantize a finished model to serve it cheaply. Quick guide to picking one:

  • GPTQ / AWQ — post-training quantization (often 4-bit) for GPU serving, with minimal quality loss. AWQ in particular is popular for fast, accurate GPU inference. Use these when serving on GPUs via engines like vLLM.
  • GGUF (via llama.cpp / Ollama) — a format optimized for CPU and Apple Silicon and easy local runs, with many quantization levels (Q4_K_M, Q5_K_M, Q8_0, …). Use it for laptops, edge, or simple self-hosting.
  • Pick a quantization level by the size/quality trade-off: 8-bit ≈ near-lossless but bigger; 4-bit ≈ great size/quality balance for most uses; below 4-bit only when memory is truly tight (quality degrades). Always re-evaluate the quantized model (Module 11) — quantization is “mostly lossless,” not “always lossless,” and your task may be sensitive.

Rule of thumb: for most self-hosted fine-tunes, a 4-bit AWQ/GPTQ (GPU) or Q4_K_M GGUF (CPU/Mac) is the default starting point. Measure, then adjust.

3. Serving engines: what actually runs the model

You rarely call model.generate() directly in production — it’s slow and handles one request at a time. You use a serving engine built for throughput. Know these:

  • vLLM — the de-facto high-performance GPU serving engine. Its key trick, PagedAttention, manages memory so it can batch many requests efficiently (continuous batching), giving far higher throughput than naive serving. Exposes an OpenAI-compatible API. Default choice for serious GPU serving, and it supports serving LoRA adapters (§5).
  • Text Generation Inference (TGI) — Hugging Face’s production server; similar goals, also widely used.
  • Ollama — dead-simple local serving of GGUF models on Mac/PC; perfect for prototypes, local apps, and small-scale self-hosting. Not for high-scale production, but wonderful for getting started.
  • llama.cpp — the lightweight engine under Ollama; runs quantized models efficiently on CPU/GPU/Mac. Great for edge and embedded.
  • Managed/cloud endpoints — cloud providers and platforms host your model behind an API so you don’t manage servers. Less control, less ops burden, higher per-token cost.

The throughput concepts that make these fast and that you should understand:

  • Batching / continuous batching — process many requests together to use the GPU fully. The single biggest lever for cost-efficiency.
  • KV cache — the model caches the attention Keys/Values (Module 01 §5a) of tokens already generated so it doesn’t recompute them each step. It speeds generation but consumes memory that grows with context length and concurrency — often the real limit on how many requests you can serve at once.

4. A minimal serving example

Getting your merged model behind an API with vLLM is roughly:

# Serve a (merged, optionally quantized) model with an OpenAI-compatible API
pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model ./acme-merged \
    --quantization awq \           # if you quantized with AWQ for serving (§2)
    --max-model-len 4096
# Call it exactly like the OpenAI API — easy to integrate
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="./acme-merged",
    messages=[
        {"role": "system", "content": "You write support replies for Acme Cloud..."},
        {"role": "user",   "content": "I was charged twice this month!"},
    ],
)
print(resp.choices[0].message.content)

For a quick local demo instead: ollama create acme -f Modelfile && ollama run acme with a GGUF build. Use Ollama to prototype, vLLM to scale.

5. Multi-LoRA serving: the superpower pattern

This is where the PEFT advantages from Module 06 §8 pay off spectacularly, and it’s worth its own section because it’s a genuine competitive edge.

Because a LoRA adapter is tiny and separate from the base (Module 07 §4), you can load one base model into GPU memory and hot-swap many adapters per request. vLLM and similar engines support this directly:

                 ┌─────────────────────────────┐
   request A ───►│                             │──► answer (Acme support voice)
   (adapter:     │   ONE base model in memory  │
    acme)        │   + swap small adapters     │
   request B ───►│   per request               │──► answer (legal-summarizer voice)
   (adapter:     │                             │
    legal)       └─────────────────────────────┘

Why this is huge:

  • Serve dozens of specialized “models” at ~the cost of one. You pay for one base in memory, plus a few MB per adapter, instead of a full model per task.
  • Per-customer or per-task customization at scale. A SaaS product can fine-tune a small adapter per customer and serve them all from shared infrastructure.
  • Instant updates. Ship a new adapter without redeploying the whole model.

This pattern — one base, many adapters, swapped at request time — is one of the most commercially important consequences of LoRA, and understanding it marks you as someone who gets the systems side, not just training.

6. The operational realities: cost, latency, scaling

Shipping is where engineering judgment matters. The levers and trade-offs:

  • Latency has two parts: time-to-first-token (how fast the user sees something — dominated by prompt length and queueing) and tokens-per-second (generation speed). Streaming tokens to the user hides latency and improves perceived speed. Smaller/quantized models and shorter prompts (a fine-tuning benefit! Module 02 §3 #5) both help.
  • Throughput vs latency trade-off. Bigger batches raise throughput (lower cost per request) but can raise individual latency. Tune batch settings to your needs.
  • Cost drivers: GPU memory (model size + KV cache), utilization (batching!), and tokens processed. Quantization, smaller fine-tuned models, multi-LoRA sharing, and shorter prompts are your main cost reducers — and notice how many of these are direct payoffs of fine-tuning a small model well.
  • Hardware sizing: estimate memory as (quantized model size) + (KV cache for your max context × expected concurrency). The KV cache surprises people — long contexts and high concurrency eat memory fast (§3).
  • Scaling: for real load you’ll run multiple replicas behind a load balancer, with autoscaling. Start simple (one box), measure, then scale what the data says to scale.

7. Production hygiene: monitoring, safety, versioning

The difference between a demo and a product is what happens after launch. Don’t skip these:

  • Monitor in production. Track latency, error rates, throughput, and — crucially — output quality drift over time. Log inputs/outputs (respecting privacy, Module 09 §7) so you can investigate failures and build your next training set from real traffic.
  • Guardrails / safety. Put input and output filters around the model (block disallowed requests, catch unsafe or PII-leaking outputs). Remember fine-tuning can weaken base safety (Module 11 §7) — verify and re-add protection at the serving layer.
  • Versioning & rollback. Version your models and the data and config that produced them (Module 10 §7). Be able to roll back instantly when a new model regresses. Treat models like code releases.
  • Evaluate before every deploy. Run your Module 11 harness (target task + regression + safety) on each candidate. Never ship a model you haven’t measured against the current one.
  • Feedback loop. Capture real user feedback (thumbs up/down, corrections). This is gold: it’s both your best evaluation signal and your best source of new training/preference data (Module 12 §3, KTO). The flywheel — deploy → collect feedback → curate → retrain → redeploy — is how production models keep improving.
  • A/B test changes. Roll new models to a fraction of traffic and compare real metrics before full rollout, rather than trusting offline evaluation alone.

8. Putting the production architecture together

For our Acme assistant (Module 13), the full production shape looks like:

   user message
        │
        ▼
   [ guardrails: input filter ]                         (§7)
        │
        ▼
   [ RAG: retrieve this customer's plan, current outage status ]   (facts — Module 02 §2)
        │
        ▼
   [ fine-tuned model served by vLLM, maybe via a per-tenant LoRA adapter ]  (behavior — §3, §5)
        │
        ▼
   [ guardrails: output filter, PII check ]             (§7)
        │
        ▼
   stream reply to user  ──►  log + collect feedback  ──►  next training set   (§7 flywheel)

Read that diagram top to bottom and notice it uses every idea in the course: the fine-tune supplies behavior, RAG supplies facts, quantization and batching supply efficiency, multi-LoRA supplies scale, guardrails supply safety, and the feedback log supplies your next iteration’s data. That integrated picture — not any single technique — is what mastery looks like.

Module 14 checklist

  • I can choose between merging and keeping the adapter separate, and why.
  • I know the serving quantization options (GPTQ/AWQ for GPU, GGUF for CPU/Mac) and to re-evaluate after.
  • I can name the main serving engines (vLLM, TGI, Ollama, llama.cpp) and when to use each.
  • I understand batching, continuous batching, and the KV cache’s memory cost.
  • I can explain multi-LoRA serving and why it’s commercially powerful.
  • I can reason about latency, throughput, and cost drivers.
  • I know the production-hygiene practices: monitoring, guardrails, versioning, feedback flywheel.

➡️ Next: Module 15 — Advanced Topics & Mastery

← Module 13: Hands-On End-to-End Project