Applied AI / Fine-Tuning & Model Customization / Module 05 · Supervised Fine-Tuning (SFT)
Module 05

Supervised Fine-Tuning (SFT)

Training & Customization Fine-Tuning & Model Customization

Module 05 — Supervised Fine-Tuning (SFT)

Goal: Master the workhorse technique of the whole field. SFT is what people usually mean by “fine-tuning.” You already have all the pieces — model (M01), the decision to do it (M02), how learning works (M03), and data (M04). Now we assemble them into a real training run, with code you can actually execute, and the judgment to make it work.

1. What SFT is, precisely

Supervised Fine-Tuning means: continue training a pretrained model on a labeled dataset of input → desired output examples, using the same next-token-prediction objective from pretraining, so the model learns to produce the desired outputs.

“Supervised” = every example has a known correct answer (the supervision) that we want the model to imitate. It’s the same machinery as pretraining (predict the next token, minimize cross-entropy loss), but on a small, focused, curated dataset instead of the whole internet.

One-line mental model: SFT teaches behavior by example. Show the model what a good response looks like, thousands of times, and it learns to generalize that behavior.

This is exactly how raw base models become helpful “instruct” assistants (Module 01, §7) — and it’s how you’ll specialize a model for your task.

2. Where SFT sits in the bigger picture

SFT is the foundation that other techniques build on:

  Base model (raw, just continues text)
        │
        ▼  SFT  ← THIS MODULE: teach it to follow instructions / do your task
  Instruct/task model (helpful, does what you trained)
        │
        ▼  Preference tuning (DPO/RLHF, Module 12)  ← optional polish: teach taste
  Aligned model (helpful AND well-judged)

You almost always do SFT first. Preference tuning (Module 12) refines a model that already behaves roughly right; it’s not a substitute for SFT. And LoRA/QLoRA (Modules 06–08) are not a different goal — they’re cheaper ways to do SFT (and DPO). So this module’s concepts carry directly into those.

3. Full fine-tuning vs parameter-efficient: a first look

There are two ways to perform SFT, distinguished by how many weights you update:

  • Full fine-tuning: update all the model’s weights. Maximum flexibility and (sometimes) maximum quality, but maximum cost — recall from Module 03 you need memory for the weights, their gradients, and the optimizer state, roughly 4×+ the model size, plus activations. Fine-tuning a 7B model this way wants a serious multi-GPU setup; a 70B model, a small cluster.
  • Parameter-efficient fine-tuning (PEFT): freeze almost all weights and train only a tiny added set. ~99% less trainable memory, runs on one consumer GPU, and — bonus — resists overfitting. This is LoRA/QLoRA, the subject of the next three modules.

For learning, this module shows the full fine-tuning mindset and a complete SFT loop; in practice you’ll almost always use LoRA/QLoRA to run it. The training concepts, data, and trainer are identical — PEFT just changes which parameters are trainable. So everything here transfers. We’ll write the code in the standard trl style that works for both, switching a couple of lines to enable LoRA in Module 07.

4. Choosing your base model

A real, consequential decision. Consider:

  1. Base vs Instruct (Module 01, §7). For most tasks, start from the Instruct version — you inherit existing instruction-following and only teach your specialization. Start from the base version when you’re imposing a very different format/behavior, want to avoid the instruct model’s existing style/guardrails, or have a lot of data.
  2. Size vs your hardware and latency needs. Bigger models are more capable but slower, costlier, and harder to fine-tune. A well-fine-tuned 7–8B model often beats a generic much larger one on your specific task — and it’s the practical sweet spot for one-GPU QLoRA. Start small; scale up only if evaluation shows you need to.
  3. License. Check the license permits your use (commercial? redistribution?). This is a real-world gotcha that bites companies.
  4. Tokenizer and language coverage. Make sure the model handles your language/domain reasonably out of the box; fine-tuning shapes existing ability more than it creates new ability (Module 02, §4).
  5. Ecosystem and context length. Popular, well-supported families (Llama, Mistral, Qwen, Gemma, Phi) have better tooling and community help — worth a lot when debugging. Confirm the context length fits your examples.

Practical default for your first real project: a popular 7–8B Instruct model, fine-tuned with QLoRA. It’s the path of least resistance with the most help available.

5. The key hyperparameters for SFT (and how to set them)

You met the concepts in Module 03; here’s how to actually choose values. These are starting points, not laws — evaluation (Module 11) decides what’s right.

HyperparameterWhat it doesSensible starting point
Learning rateStep size (M03 §4). Most important knob.1e-4–2e-4 for LoRA; 1e-5–2e-5 for full FT
EpochsPasses over the data (M03 §5).1–3. Watch validation loss; stop when it bottoms out
Batch size (per_device_train_batch_size)Examples per step.As many as fit in memory (often 1–8 on one GPU)
Gradient accumulationSimulates a bigger batch (M03 §5).Set so effective batch = 16–64
Max sequence lengthToken cap per example (M04 §4).Smallest that fits your real examples (e.g., 1024–2048)
WarmupRamp LR up at the start (M03 §4).3–10% of total steps, or a few hundred steps
LR scheduleHow LR decays.cosine is a fine default
Weight decayRegularization vs overfitting.0.0–0.1 (e.g., 0.01)
OptimizerHow steps are applied (M03 §6).adamw_8bit / paged_adamw_8bit to save memory
PrecisionNumerical format for speed/memory.bf16 if your GPU supports it, else fp16

How to actually tune them, in order of impact:

  1. Get learning rate in the right ballpark first — it dominates everything. If loss is unstable or NaN, lower it.
  2. Set epochs by the validation curve, not a fixed number. Overfitting (M03 §7) is the enemy.
  3. Use gradient accumulation to reach a stable effective batch size if memory is tight.
  4. Touch the rest only if evaluation suggests a problem. Don’t fiddle with ten knobs at once (M00 §6).

6. A complete SFT script (read every line)

Here’s a real, runnable SFT using Hugging Face trl’s SFTTrainer. This is the full fine-tuning–style setup; Module 07 adds ~6 lines to make it LoRA. Comments explain the why.

# pip install transformers datasets trl accelerate
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig

# 1) Pick a base model (an Instruct model is a good default — Module 05 §4)
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"   # example; swap as needed

# 2) Load tokenizer (MUST match the model — Module 04 §3)
tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token   # ensure a padding token exists

# 3) Load the model
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,   # bf16 for memory/speed (Module 05 §5)
    device_map="auto",
)

# 4) Load your data as chat-format messages (Module 04 §2b) and split
ds = load_dataset("json", data_files="my_data.jsonl", split="train")
ds = ds.train_test_split(test_size=0.1, seed=42)

# 5) Configure training (your hyperparameters from Module 05 §5)
config = SFTConfig(
    output_dir="./sft-output",
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,     # effective batch = 2 * 8 = 16
    learning_rate=2e-5,                # full-FT range; LoRA would use ~2e-4
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    weight_decay=0.01,
    bf16=True,
    logging_steps=10,                  # print loss often so you can WATCH it
    eval_strategy="steps",
    eval_steps=50,                     # check validation loss regularly (overfitting!)
    save_strategy="steps",
    save_steps=50,
    max_seq_length=2048,               # Module 04 §4
    packing=True,                      # pack short examples for efficiency
)

# 6) The trainer wires it all together (it applies the chat template &
#    completion-only masking for you — Module 04 §3, §5)
trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    processing_class=tokenizer,
)

# 7) BEFORE training: inspect ONE fully-formatted example (Module 04 §8)!
print(tokenizer.apply_chat_template(ds["train"][0]["messages"], tokenize=False))

# 8) Train. Watch training AND validation loss in the logs.
trainer.train()

# 9) Save
trainer.save_model("./sft-final")
tokenizer.save_pretrained("./sft-final")

Things to actually do while it runs:

  • Watch the two losses in the logs. Training loss should fall; validation loss should fall then flatten. If validation rises while training keeps dropping → overfitting → stop / reduce epochs (M03 §7).
  • If you hit out-of-memory, your first levers: lower per_device_train_batch_size (raise gradient_accumulation_steps to compensate), lower max_seq_length, enable gradient checkpointing, or switch to QLoRA (Module 08). OOM is the most common error you’ll meet; these are the standard responses.

7. Catastrophic forgetting: the side effect to respect

Recall Module 02 warned that fine-tuning can make a model worse at things you didn’t train on. The name is catastrophic forgetting: as weights shift to fit your narrow data, they drift away from the broad competence pretraining gave them. Train an instruct model hard on 2,000 terse customer-support replies and it may become great at support but worse at general reasoning or other styles.

How to keep it in check:

  • Train gently: fewer epochs, lower learning rate, just enough to get the behavior.
  • Use LoRA/PEFT: because it changes far fewer weights, it perturbs the base model less and forgets less — a real, underappreciated advantage of the methods in the next modules.
  • Mix in some general/diverse data alongside your task data so the model keeps its broad skills (sometimes called “replay” or “data mixing”).
  • Evaluate on both your task and general capabilities (Module 11), so you actually notice forgetting instead of being surprised in production.

Expert framing: fine-tuning is a trade. You spend some general capability to buy specialized capability. Good practitioners spend deliberately and measure the bill.

8. Common SFT failure modes and their fixes

A field guide you’ll return to. Most fine-tuning problems are one of these:

  • The model never stops generating / rambles. Almost always a chat-template or end-of-sequence-token problem (M04 §3). Ensure the template is applied and the EOS token is present so the model learns where answers end.
  • Loss goes to NaN or spikes wildly. Learning rate too high (M03 §4) → lower it. Or precision instability → try bf16, add warmup, clip gradients.
  • Loss barely moves (underfitting). LR too low, too few epochs, frozen weights by mistake, or data too noisy. Train more/harder; check the data.
  • Great on training data, bad on new inputs (overfitting). Fewer epochs / early stopping, more or more-diverse data, add weight decay/dropout, or use LoRA (M03 §7).
  • Model outputs the user’s text or weird artifacts. Masking misconfigured (M04 §5) or wrong template. Inspect a formatted example.
  • Model became worse at general tasks. Catastrophic forgetting (§7) — train more gently, mix in general data, prefer LoRA.
  • Output format is inconsistent. Your data was inconsistent (M04 §1). Fix the dataset, not the hyperparameters — this is the most common real cause and the hardest for beginners to accept.

Notice how many fixes point back to data and template, not exotic settings. That’s the recurring lesson of the whole course.

9. Your SFT mental checklist (use before every run)

A pre-flight list the pros run, explicitly or by reflex:

  1. Right base model (base vs instruct), license OK, fits hardware?
  2. Data in the right format, cleaned, consistent, split without leakage?
  3. Inspected one fully-formatted, tokenized example? (Non-negotiable.)
  4. Chat template correct; masking doing what I intend?
  5. Hyperparameters sane (LR ballpark, 1–3 epochs, effective batch reasonable)?
  6. Validation set wired up so I can see overfitting?
  7. A plan to evaluate the result beyond eyeballing (Module 11)?
  8. Notebook entry written: model, data, key settings, hypothesis?

If all eight are yes, hit train. If not, the cheapest time to fix a problem is before you spend an hour of GPU time.

Module 05 checklist

  • I can define SFT and explain how it relates to pretraining and to DPO/RLHF.
  • I understand full FT vs PEFT and that LoRA is just a cheaper way to do SFT.
  • I can choose a base model deliberately (base vs instruct, size, license).
  • I can set the core hyperparameters and know which to tune first.
  • I can read the SFT script and say what each step does.
  • I can explain catastrophic forgetting and how to limit it.
  • I can diagnose the common SFT failure modes from their symptoms.

➡️ Next: Module 06 — PEFT: Parameter-Efficient Fine-Tuning

← Module 04: Data Foundations: The Fuel of Fine-Tuning