Module 12 — Preference Tuning: RLHF, DPO & Friends

Goal: Understand the second great fine-tuning technique — teaching a model taste and judgment, not just facts and formats. You’ll learn why SFT alone isn’t enough for the best behavior, how RLHF works conceptually, why DPO largely replaced it for most practitioners, and how to actually run preference tuning. This is what turns a competent model into a genuinely good one.

1. The limitation of SFT that motivates this whole module

SFT (Module 05) teaches by imitation: “here’s a good answer, copy this style.” It’s powerful, but it has a blind spot:

SFT can only show the model what to do. It can’t easily show the model what not to do, or which of two decent answers is better.

Many qualities we care about are comparative and subjective: helpfulness, harmlessness, honesty, tone, conciseness, “did it actually answer the question?” These are hard to capture as single gold examples but easy to capture as preferences: given two responses, this one is better.

That’s the core idea of preference tuning:

Instead of (or after) showing the model ideal answers, we show it pairs of answers labeled “this one is better than that one,” and train it to produce more of the preferred kind.

This is exactly how raw models became the polished assistants you know (Module 01 §7): SFT first to make them helpful, then preference tuning to refine their judgment.

2. Where preference tuning sits

It comes after SFT, as a refinement — never instead of it:

  Base model
      │ SFT (Module 05) — teach it to follow instructions / do your task
      ▼
  SFT model (competent, roughly right behavior)
      │ Preference tuning (THIS MODULE) — teach it taste/judgment
      ▼
  Aligned model (helpful, harmless, honest, polished)

You preference-tune a model that already behaves roughly right. Trying to preference-tune a model that hasn’t been SFT’d is like teaching nuance to someone who hasn’t learned the basics — it doesn’t work well. SFT establishes competence; preference tuning establishes quality.

3. The preference data format

Recall from Module 04 §2c. Each example is a prompt with a chosen (better) and rejected (worse) response:

{
  "prompt": "Explain why the sky is blue to a curious 8-year-old.",
  "chosen":   "Sunlight is made of all colors mixed together. When it hits the air, the blue light bounces around the most, so the sky looks blue!",
  "rejected": "The phenomenon results from Rayleigh scattering, wherein shorter wavelengths of electromagnetic radiation are scattered more efficiently by atmospheric particulates."
}

Both answers may be correct; the point is one is better for the goal (here: explaining to a child). Where does this data come from?

Human preference labels — show people two model outputs, ask which is better (Module 11 §5). High quality, expensive.
AI feedback — use a strong model to pick the better of two (RLAIF / LLM-as-judge, Module 11 §6). Cheaper, scalable, with the biases you learned to watch for.
Constructed pairs — e.g., chosen = a known-good answer, rejected = a deliberately worse/older-model/violating one.

The quality and consistency of these preferences decides everything (Module 10 applies fully here).

4. RLHF: the original approach (understand it, even if you won’t run it)

RLHF — Reinforcement Learning from Human Feedback — was the breakthrough method behind the first great chat assistants. It’s more involved than SFT, so let’s build the intuition; you should understand it conceptually even though DPO (§5) is what you’ll usually run.

RLHF has two stages:

Stage 1 — Train a reward model

Take all those “chosen vs rejected” preference pairs and train a separate model — the reward model — whose job is to score any response with a number representing how good humans would find it. It learns from the pairs: give the chosen response a higher score than the rejected one. Now you have an automatic “taste meter.”

Stage 2 — Optimize the LLM against the reward model with RL

Now use reinforcement learning (typically an algorithm called PPO — Proximal Policy Optimization) to update the language model so it generates responses the reward model scores highly. The loop:

  LLM generates a response  ──►  reward model scores it  ──►  RL nudges the LLM
        ▲                                                     to get higher scores
        └─────────────────────  repeat  ◄─────────────────────────────────────┘

One crucial safeguard: a KL-divergence penalty keeps the LLM from drifting too far from the original SFT model while chasing reward. Without it, the model finds degenerate ways to game the reward model (producing weird text that scores high but is actually bad — reward hacking). The penalty is a leash that says “improve, but stay recognizably yourself.”

Why RLHF is hard: it requires training and hosting a separate reward model, running a finicky RL algorithm, juggling multiple models in memory at once, and tuning unstable hyperparameters. It’s powerful but heavy, expensive, and tricky to get stable. That difficulty is exactly why the next method took over.

5. DPO: the method most people actually use now

DPO — Direct Preference Optimization — was a landmark simplification (2023). Its insight is beautiful:

You can skip the separate reward model and the reinforcement learning entirely. There’s a mathematical shortcut that lets you optimize the language model directly on the preference pairs, with a simple training loss that looks much like ordinary SFT.

In other words, DPO achieves the goal of RLHF — “make preferred responses more likely, dispreferred ones less likely, without drifting too far from the SFT model” — but with a single model and a straightforward training loop, no reward model, no PPO. The KL “stay close to the original” idea is baked right into its loss (via a reference copy of the SFT model and a parameter called beta that controls how strongly to stay close).

The practical upshot:

Far simpler and more stable than RLHF — if you can run SFT, you can run DPO.
Cheaper — fewer models, no RL machinery.
Often comparable quality to RLHF for most use cases.

This is why, for the overwhelming majority of practitioners, DPO (or one of its cousins) is the default way to do preference tuning today. RLHF remains relevant at the frontier and where its extra power is needed, but DPO is your starting point.

6. DPO in code

The wonderful part: DPO in trl looks almost exactly like the SFT you already know (Module 05/07), and it combines with LoRA/QLoRA just the same. Differences: a DPOTrainer, preference-format data, and a beta parameter.

# pip install transformers datasets trl peft
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import DPOTrainer, DPOConfig

# 1) Start from your SFT'd model (preference tuning refines an SFT model — §2)
model_name = "./sft-final"           # the model you produced in Module 05/07
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 2) Preference data: prompt / chosen / rejected (§3, Module 04 §2c)
ds = load_dataset("json", data_files="preferences.jsonl", split="train")
ds = ds.train_test_split(test_size=0.1, seed=42)

# 3) LoRA — same as ever (Module 07); DPO works great with PEFT
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                         lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")

# 4) DPO config — note beta (how strongly to stay near the reference model, §5)
config = DPOConfig(
    output_dir="./dpo-out",
    num_train_epochs=1,              # preference tuning usually needs FEW epochs
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-6,              # typically LOWER than SFT — gentle refinement
    lr_scheduler_type="cosine", warmup_ratio=0.1,
    beta=0.1,                        # the KL "stay close" strength (0.1 is common)
    bf16=True, logging_steps=10,
    eval_strategy="steps", eval_steps=50,
)

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    processing_class=tokenizer,
    peft_config=lora_config,         # DPO + LoRA, just like SFT + LoRA
)
trainer.train()
trainer.save_model("./dpo-adapter")

Things to note, all consequences of what you’ve learned:

Start from the SFT model, not the raw base (§2).
Lower learning rate and fewer epochs than SFT — this is gentle refinement, and over-doing it degrades the model fast (preference tuning overfits and “collapses” easily — watch evaluation closely, Module 11).
beta is the key new knob: higher = stay closer to the SFT model (safer, less change); lower = allow bigger shifts toward preferences (more change, more risk). Start around 0.1.
It composes with QLoRA identically (add the BitsAndBytesConfig from Module 08) — that’s how people run DPO on a single GPU.

7. The DPO family (recognize the names)

DPO spawned a family of simplified preference methods. You don’t need them to start, but fluency means recognizing them:

DPO — the original; needs a frozen reference copy of the SFT model.
IPO — a variant that changes the objective to be more robust to overfitting on the preferences.
KTO (Kahneman-Tversky Optimization) — needs only a label of “good” or “bad” per single response, not paired chosen/rejected. Useful when you can’t easily produce pairs but can say “this output was thumbs-up / thumbs-down” — which matches a lot of real product feedback.
ORPO — cleverly combines SFT and preference tuning into one step (no separate SFT stage, no reference model needed). Attractive for simplicity and efficiency.
SimPO, others — further refinements removing the reference model or tweaking the loss.

The common thread — the durable understanding to keep — is that they all learn from preference/quality signals to push good behavior up and bad behavior down, more directly and cheaply than classic RLHF. Master DPO’s intuition and you can pick up any of them quickly.

8. When do you actually need preference tuning?

Be deliberate — it adds complexity, so use it when it earns its keep:

Reach for it when:

SFT got the model competent but you need to refine quality — tone, helpfulness, conciseness, safety, “answers the actual question.”
You have, or can cheaply create, preference data (human or AI).
You’re reducing specific bad behaviors (rambling, hedging, unsafe outputs) that are easier to define as “worse than” than to demonstrate perfectly.

Skip it (for now) when:

You haven’t nailed SFT yet — do that first (§2).
A pure imitation task (classification, extraction, format conversion) is already solved well by SFT — preference tuning adds little.
You don’t have quality preference data and can’t generate trustworthy pairs.

Practical path for most projects: get SFT + good data + solid evaluation working first. Add DPO as a second pass when evaluation (Module 11) shows your model is competent but its judgment/quality needs polishing. Many excellent fine-tuned models never need more than SFT; preference tuning is the finishing move, not the foundation.

Module 12 checklist

I can explain why SFT alone can’t easily teach “better vs worse” and what preferences add.
I know preference tuning comes after SFT, as refinement.
I can describe the preference data format and where it comes from.
I can explain RLHF’s two stages (reward model + PPO) and the KL leash / reward hacking.
I can explain DPO’s insight (skip the reward model and RL) and why it’s the common default.
I can run DPO with LoRA, and I know beta, lower LR, and fewer epochs matter.
I can name DPO-family methods (IPO, KTO, ORPO) and when preference tuning is worth it.

➡️ Next: Module 13 — Hands-On End-to-End Project