Module 13 — Hands-On End-to-End Project

Goal: Tie everything together by building a real fine-tuned model from scratch — the full loop from Module 00, with actual decisions and code. We’ll build a customer-support assistant for a fictional company, “Acme Cloud.” You’ll make every choice deliberately, using the reasoning from the earlier modules. Do this once, all the way through, and the abstract becomes concrete.

1. The project and the plan

The task: Fine-tune a small, self-hostable model to answer Acme Cloud customer-support questions in the company’s voice — concise, friendly, accurate, and always ending with a next step. We want it cheap to run (no frontier-model API bills) and consistent.

Why fine-tuning is right here (Module 02 decision framework):

We tried prompting; a good prompt gets close but the tone drifts and replies ramble. → behavior/consistency problem ✔
The general support style is stable behavior, not changing facts. → fine-tuning, not RAG, for the style ✔
Specific, current facts (a customer’s plan, today’s outage status) should come from RAG at inference, combined with our fine-tune. → we’ll note where RAG plugs in ✔
We want a cheap, private, self-hosted model. → classic fine-tuning win (Module 02 §3 #6) ✔

The plan (the loop from Module 00 §2):

Get data (we have few real tickets → generate synthetic, Module 09).
Curate it (Module 10).
Fine-tune with QLoRA (Modules 05/07/08).
Evaluate against the base model (Module 11).
Optionally refine with DPO (Module 12).
Merge & prepare for serving (Module 14 next).

2. Step 1 — Data: generate and curate

We have ~40 real support tickets — not enough (Module 04 §6). We’ll generate synthetic data grounded in them (Module 09), then curate hard (Module 10).

import json, random
from openai import OpenAI
client = OpenAI()

# Diversity seeding (Module 09 Pattern F): vary scenarios so outputs aren't clones
scenarios = [
    "can't log in after a password reset",
    "wants to upgrade from Free to Pro plan",
    "API returning 429 rate-limit errors",
    "billing charged twice this month",
    "how to invite a teammate",
    "data export is stuck / slow",
    "wants to cancel and asks about refunds",
    "SSO / SAML setup question from an admin",
    # ... 20-40 varied scenarios; the more variety, the better ...
]

# Brand voice spec — bake the behavior we want into the generator (Module 09 §2)
SYSTEM_SPEC = (
    "You write support replies for Acme Cloud. Voice: warm, concise (max 4 sentences), "
    "plain language, no jargon unless the user used it. ALWAYS end with a concrete next step. "
    "If you don't know a specific account fact, say you'll check rather than inventing it."
)

gen_prompt = """Scenario: {scenario}
Produce ONE training example as strict JSON:
{{"messages":[
  {{"role":"system","content":"{spec}"}},
  {{"role":"user","content":"<a realistic, natural customer message for this scenario>"}},
  {{"role":"assistant","content":"<an ideal Acme Cloud reply following the voice spec>"}}
]}}"""

raw = []
for scenario in scenarios:
    for _ in range(40):                     # ~40 per scenario → generate generously
        r = client.chat.completions.create(
            model="a-strong-teacher-model",  # distillation (Module 09 Pattern A)
            messages=[{"role":"user","content": gen_prompt.format(scenario=scenario, spec=SYSTEM_SPEC)}],
            temperature=0.9,                 # diversity (watch quality)
        )
        try: raw.append(json.loads(r.choices[0].message.content))
        except json.JSONDecodeError: pass    # drop malformed (Module 10 Stage 3)

print(f"Generated {len(raw)} raw examples")

Now curate (Module 10) — this is where quality is won:

import hashlib

def assistant_text(ex): return ex["messages"][-1]["content"]
def user_text(ex):      return ex["messages"][-2]["content"]

clean, seen = [], set()
for ex in raw:
    a = assistant_text(ex)
    # Stage 3 filters (Module 10 §3): format + quality rules
    if not a or len(a) < 15:                     continue   # too short / empty
    if a.count(".") > 6:                          continue   # likely rambling (>4-sentence spec)
    if "as an ai" in a.lower():                   continue   # boilerplate leak
    # Stage 4 dedup (Module 10 §4): exact dedup on the (user,assistant) pair
    key = hashlib.md5((user_text(ex)+a).encode()).hexdigest()
    if key in seen:                               continue
    seen.add(key)
    clean.append(ex)

print(f"Kept {len(clean)} / {len(raw)} after curation")   # expect to drop 20-60% (Module 09 §5)

# Module 10 §3 Stage 1: ALWAYS read a sample yourself
for ex in random.sample(clean, 5):
    print(user_text(ex), "→", assistant_text(ex), "\n")

# Reserve a REAL-only test set (Module 09 §6, Module 11 §2) — keep your 40 real tickets out of training!
with open("acme_train.jsonl","w") as f:
    for ex in clean: f.write(json.dumps(ex)+"\n")

Notice we did not put our 40 real tickets into training — we hold them as a real-only test set (Module 11 §2) so evaluation measures real-world performance, not how well we imitate the generator.

3. Step 2 — Fine-tune with QLoRA

We’ll fine-tune an 8B instruct model with QLoRA so it runs on a free/cheap GPU (Module 08). This is the Module 08 script applied to our data.

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig

model_name = "meta-llama/Meta-Llama-3-8B-Instruct"   # 8B instruct: good sweet spot (Module 05 §4)

# QLoRA 4-bit base (Module 08 §3)
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)
tok = AutoTokenizer.from_pretrained(model_name)
if tok.pad_token is None: tok.pad_token = tok.eos_token

# Data + split (Module 04 §7)
ds = load_dataset("json", data_files="acme_train.jsonl", split="train").train_test_split(test_size=0.1, seed=42)

# SANITY CHECK — inspect one fully-formatted example (Module 04 §8). Do NOT skip.
print(tok.apply_chat_template(ds["train"][0]["messages"], tokenize=False))

lora = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                  lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")   # Module 07 §5

cfg = SFTConfig(
    output_dir="./acme-qlora",
    num_train_epochs=2,                      # 1-3 epochs (Module 05 §5); watch validation
    per_device_train_batch_size=2, gradient_accumulation_steps=8,   # eff. batch 16 (Module 03 §5)
    learning_rate=2e-4,                      # LoRA range (Module 07 §6)
    lr_scheduler_type="cosine", warmup_ratio=0.05,
    optim="paged_adamw_8bit", gradient_checkpointing=True,          # memory (Module 08 §5)
    bf16=True, logging_steps=10,
    eval_strategy="steps", eval_steps=25,    # watch overfitting (Module 03 §7)
    max_seq_length=1024, packing=True,
)

trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds["train"],
                     eval_dataset=ds["test"], processing_class=tok, peft_config=lora)
trainer.train()                              # WATCH train vs validation loss as it runs
trainer.save_model("./acme-adapter")         # tiny adapter (Module 06 §8)

What to watch while it trains (everything you learned in Module 03):

Training loss falling, validation loss falling then flattening → good.
Validation loss rising while training falls → overfitting → stop / reduce epochs (Module 03 §7). With only ~25 eval steps visible, this is easy to catch early.
Out-of-memory? Run the Module 08 §5 checklist: lower batch size (raise accumulation), lower max_seq_length, ensure gradient checkpointing on.

4. Step 3 — Evaluate against the base model

Never trust vibes (Module 11). Compare fine-tuned vs base on our real test set, on the criteria we actually defined.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import json

tok = AutoTokenizer.from_pretrained(model_name)
base = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto")
ft   = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto"),
    "./acme-adapter")

real_test = [json.loads(l) for l in open("acme_real_test.jsonl")]   # the held-out REAL tickets

def reply(m, messages):
    ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(m.device)
    out = m.generate(ids, max_new_tokens=200, do_sample=False)
    return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)

# Our concrete criteria (Module 11 §4b): concise (<=4 sentences), ends with a next step, on-topic.
def score(text):
    concise   = text.count(".") <= 4
    next_step = any(w in text.lower() for w in ["next","go to","click","try","check","reply","let me"])
    return int(concise) + int(next_step)            # 0-2 quick automatic proxy

for name, m in [("base", base), ("finetuned", ft)]:
    s = sum(score(reply(m, ex["messages"][:-1])) for ex in real_test) / len(real_test)
    print(f"{name}: avg criteria score {s:.2f} / 2")

Then do what automatic proxies can’t (Module 11 §5–6):

Blinded A/B: for ~30 test prompts, put base vs fine-tuned answers side by side (order randomized) and judge which is better — by hand and/or with an LLM judge using our voice rubric. Report the win rate.
Regression check (Module 11 §7): ask both models some general questions (unrelated to support) to confirm the fine-tune didn’t get noticeably dumber (catastrophic forgetting, Module 05 §7).
Read the failures (Module 11 §9): the worst fine-tuned answers tell you exactly what data to add — maybe we have too few “refund” or “SSO” examples. Generate more of those (Module 09 §8) and loop.

This compare-and-read-failures step is the heart of professional iteration. Most of your real improvement will come from going around this loop two or three times, not from any single clever setting.

5. Step 4 (optional) — Refine quality with DPO

Suppose evaluation shows the model is competent but sometimes too formal or occasionally forgets the next-step. That’s a quality/judgment issue — exactly DPO’s job (Module 12).

Build preference pairs cheaply: for some prompts, generate two replies, and label the one that better matches the voice as chosen, the other rejected (use an LLM judge with the rubric, or quick human picks). Then run the Module 12 §6 DPO script starting from ./acme-adapter’s merged model, with low LR, beta≈0.1, one epoch — watching evaluation closely so you don’t over-tune. Re-evaluate against the SFT-only model to confirm DPO actually helped (it doesn’t always; measure).

For many projects you’ll find SFT + good data + iteration is enough and DPO is an optional polish (Module 12 §8). Don’t add it reflexively.

6. Step 5 — Prepare for serving

Once you’re happy, prepare the model for deployment (full treatment in Module 14):

from peft import PeftModel
from transformers import AutoModelForCausalLM

# Merge the adapter into a full-precision base (Module 07 §7, Module 08 §7)
base = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto")
merged = PeftModel.from_pretrained(base, "./acme-adapter").merge_and_unload()
merged.save_pretrained("./acme-merged")        # a standalone deployable model
tok.save_pretrained("./acme-merged")
# Next (Module 14): quantize for serving (GPTQ/AWQ/GGUF) and host with vLLM/Ollama,
# and plug in RAG for live account/outage facts.

And here’s where the architecture comes together: the fine-tuned model supplies the behavior (Acme voice, conciseness, next-step habit), and RAG supplies the facts (this customer’s plan, current outage status) at inference time (Module 02 §2). That combination — fine-tune for form, retrieve for facts — is the production-grade design.

7. What you just did (and why it matters)

Step back. You took a generic model and, with deliberate choices at every stage, turned it into a specialized, self-hostable assistant:

Decided fine-tuning was right and where RAG belongs (Module 02).
Built data via grounded synthetic generation and ruthless curation (Modules 04, 09, 10).
Trained efficiently with QLoRA on modest hardware (Modules 03, 05, 07, 08).
Proved it was better than baseline and checked for regressions (Module 11).
Optionally refined judgment with DPO (Module 12).
Prepared it for real serving and designed the fine-tune + RAG split (Module 14).

That is the entire professional workflow. Everything else is variations on this loop with bigger models, more data, and more rigorous evaluation. If you actually run this project end to end — even on a tiny scale — you’ll have done what most people only read about, and the rest of the field will feel like familiar territory.

Module 13 checklist

I framed the problem with the Module 02 decision framework and located RAG vs fine-tuning.
I generated grounded synthetic data and curated it (kept a real-only test set).
I ran a QLoRA fine-tune and watched the loss curves correctly.
I evaluated against the base model on real data and read the failures.
I understand when DPO would help and how to add it.
I merged the adapter and understand how fine-tune + RAG combine in production.

➡️ Next: Module 14 — Deployment & Serving