Applied AI / Fine-Tuning & Model Customization / Module 08 · QLoRA & Quantization
Module 08

QLoRA & Quantization

Training & Customization Fine-Tuning & Model Customization

Module 08 — QLoRA & Quantization

Goal: Understand quantization (compressing a model so it fits in less memory) and QLoRA (LoRA fine-tuning on top of a quantized model), the combination that lets you fine-tune a 7B–70B model on a single consumer or rental GPU. This is the technique most people actually run, so we’ll make it click.

1. The problem: even with LoRA, the base model is huge

LoRA (Module 07) made the trainable part tiny — you only train small A/B matrices. But there’s a cost LoRA doesn’t remove: the frozen base model still has to sit in GPU memory so it can run the forward pass. A 7B model in standard 16-bit precision is ~14 GB just to hold the weights, before any activations. A 13B model is ~26 GB; a 70B model ~140 GB. That alone blows past a free Colab GPU (~16 GB) or a typical consumer card.

So the question becomes: can we shrink the frozen base model itself, so it takes far less memory, without ruining it? The answer is quantization, and pairing it with LoRA gives QLoRA.

2. What quantization is (plain version)

Every weight in a model is a number stored with some precision — how many bits are used to represent it. More bits = more exact, but more memory.

  • FP32 (32-bit float): very precise, 4 bytes per weight. The “full” precision.
  • FP16 / BF16 (16-bit float): half the memory, 2 bytes per weight. The common training/inference default. (BF16 trades a little precision for a wider range and is more stable; prefer it when supported.)
  • INT8 (8-bit integer): 1 byte per weight.
  • INT4 / NF4 (4-bit): half a byte per weight. The QLoRA sweet spot.

Quantization is the process of converting weights from high precision (say 16-bit) to low precision (say 4-bit), so the model takes far less memory.

Analogy: it’s like saving a photo as a smaller file. A 4-bit model is a “compressed” version of the model — slightly less detailed, but mostly the same picture, taking a quarter of the space of 16-bit.

The headline result: a 7B model that needs ~14 GB in 16-bit fits in roughly ~4–5 GB in 4-bit. Suddenly it fits on a free GPU — with room to spare for fine-tuning.

Why doesn’t 4-bit destroy the model?

You’d expect crushing each number into 4 bits to wreck quality. It mostly doesn’t, for two reasons:

  1. Neural networks are remarkably robust to small weight errors — they have lots of redundancy, and a little noise per weight averages out.
  2. Modern quantization is clever, not naive. QLoRA introduced a special 4-bit format called NF4 (NormalFloat4), designed to match how neural-network weights are actually distributed (clustered near zero), so the limited bits are spent where the weights actually are. It also uses tricks like double quantization (quantizing the quantization constants too) and block-wise quantization (scaling small chunks separately) to preserve accuracy. The net effect: 4-bit base models keep nearly all their capability.

3. QLoRA: the key insight

Here’s the elegant idea that makes QLoRA work, and why it’s more than “just quantize then LoRA”:

Keep the big base model frozen and quantized to 4-bit (tiny memory). Add LoRA adapters in full precision (16-bit) on top, and train only those. During the forward/backward pass, the 4-bit weights are temporarily de-quantized to compute with, but never stored in high precision and never updated.

So you get:

  • The base model’s memory slashed ~4× by 4-bit quantization.
  • The trainable params already tiny, thanks to LoRA.
  • The gradients/optimizer state only for the small LoRA adapters (Module 03 §3, §6) — and crucially, you never need gradients for the frozen 4-bit base.

The combination is what makes the famous claim true: you can fine-tune a 65B-parameter model on a single 48 GB GPU, or a 7B model on a free ~16 GB Colab GPU. QLoRA (2023) was a landmark precisely because it collapsed the hardware requirement by an order of magnitude.

   ┌───────────────────────────────────────────────┐
   │  FROZEN BASE MODEL  — quantized to 4-bit (NF4) │  ← ~4x smaller, not trained
   │  (de-quantized on the fly only to compute)     │
   └───────────────────────────────────────────────┘
                         +
   ┌───────────────────────────────────────────────┐
   │  LoRA ADAPTERS  — full precision, tiny, TRAINED│  ← all the learning happens here
   └───────────────────────────────────────────────┘

A couple of supporting tricks QLoRA contributed, worth knowing by name:

  • Paged optimizers — when a memory spike would cause an out-of-memory crash, optimizer state is temporarily moved to CPU RAM (like the OS paging memory to disk), preventing crashes on long sequences. You’ll see optim="paged_adamw_8bit".
  • Double quantization and NF4 as above.

4. QLoRA in code

The beautiful part: QLoRA is your LoRA script (Module 07 §6) plus a quantization config when loading the model. Two new pieces:

# pip install transformers datasets trl peft accelerate bitsandbytes
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig

model_name = "meta-llama/Meta-Llama-3-8B-Instruct"

# 1) NEW: describe how to quantize the base model to 4-bit (NF4 + double quant)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # the NormalFloat4 format (§2)
    bnb_4bit_use_double_quant=True,       # double quantization (§3)
    bnb_4bit_compute_dtype=torch.bfloat16 # de-quantize to bf16 for the actual math
)

# 2) Load the model IN 4-BIT using that config
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,       # ← the QLoRA magic
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)   # small setup for stable 4-bit training

tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 3) LoRA config — IDENTICAL to Module 07
lora_config = LoraConfig(
    r=16, lora_alpha=32, target_modules="all-linear",
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)

# 4) Training config — note the paged optimizer
config = SFTConfig(
    output_dir="./qlora-out",
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    lr_scheduler_type="cosine", warmup_ratio=0.05,
    bf16=True, logging_steps=10,
    eval_strategy="steps", eval_steps=50,
    optim="paged_adamw_8bit",             # paged + 8-bit optimizer to save memory (§3)
    gradient_checkpointing=True,          # trade compute for memory (§5)
    max_seq_length=2048, packing=True,
)

trainer = SFTTrainer(
    model=model, args=config,
    train_dataset=train_ds, eval_dataset=val_ds,
    processing_class=tokenizer, peft_config=lora_config,
)
trainer.train()
trainer.save_model("./qlora-adapter")     # again, a tiny adapter

Compare to Module 07: the only conceptual additions are the BitsAndBytesConfig, loading load_in_4bit, prepare_model_for_kbit_training, and the paged optimizer. Everything else — data, LoRA settings, the trainer — is what you already know. QLoRA = LoRA + a 4-bit base. That’s genuinely the whole idea.

5. Other memory-saving levers (your OOM toolkit)

When you still hit out-of-memory (the most common error in this whole field), here is the standard toolkit, roughly in order of “try first.” You now understand why each works:

  1. Reduce per_device_train_batch_size (fewer examples in memory at once) and raise gradient_accumulation_steps to keep the effective batch size (Module 03 §5). First thing to try.
  2. Reduce max_seq_length — memory grows with sequence length, often steeply (Module 04 §4). If your data doesn’t need 2048 tokens, use 1024.
  3. Enable gradient_checkpointing — instead of storing all intermediate activations for the backward pass, recompute them on the fly. Saves a lot of memory at the cost of ~20–30% more compute time. Almost always worth it for big models.
  4. Use a paged/8-bit optimizer (paged_adamw_8bit) — smaller optimizer state, survives spikes (§3, Module 03 §6).
  5. Quantize harder / use QLoRA if you weren’t already (this module).
  6. Lower the rank r slightly (Module 07 §5a) — minor, but it reduces adapter + optimizer memory.
  7. Use packing to avoid wasting memory on padding (Module 04 §4).

Memorize this list. “The model won’t fit” is not a wall; it’s a checklist.

So far we quantized to train (QLoRA). Quantization is also used to deploy models cheaply — running a finished model in 4-bit or 8-bit so it fits on smaller/cheaper hardware and runs faster. You’ll meet this again in Module 14. Key names you’ll encounter:

  • GPTQ and AWQ — popular post-training quantization methods that compress a finished model for fast inference, often with very little quality loss. Great for serving.
  • GGUF — a file format (used by llama.cpp and Ollama) for running quantized models efficiently, including on CPUs and Macs. The format behind a lot of “run an LLM on your laptop” tooling.
  • Quantization levels like Q4_K_M, Q5_K_M, Q8_0 — different trade-offs of size vs quality you’ll see when downloading models.

The distinction to keep straight:

  • QLoRA = quantize the base to fine-tune cheaply (then you typically train an adapter, and often merge + re-quantize for serving).
  • GPTQ/AWQ/GGUF = quantize a finished model to serve cheaply.

Both rest on the same core idea you now understand: fewer bits per weight, cleverly chosen, mostly preserves the model.

7. A subtle but important deployment note

After QLoRA training you have a 4-bit base + a 16-bit LoRA adapter. To deploy, you generally merge the adapter into a full-precision base, then (optionally) re-quantize the merged model for serving (Module 07 §7, Module 14). Merging directly into the 4-bit base can lose quality, because the adapter was trained against an approximation. The clean recipe most pros follow:

  1. Train with QLoRA (4-bit base + LoRA).
  2. Reload the base in 16-bit, attach the adapter, merge_and_unload().
  3. Save the merged 16-bit model.
  4. Quantize that (GPTQ/AWQ/GGUF) for efficient serving if needed.

Knowing this avoids a subtle quality leak that trips up many first deployments.

Module 08 checklist

  • I can explain quantization as “fewer bits per weight to save memory.”
  • I know the precisions (FP32/16, INT8, NF4/4-bit) and roughly the memory each costs.
  • I can explain why 4-bit doesn’t ruin a model (robustness + NF4/double quant).
  • I can state the QLoRA insight: frozen 4-bit base + trainable 16-bit LoRA.
  • I can turn a LoRA script into QLoRA by adding a quantization config.
  • I can list the OOM toolkit and why each item helps.
  • I know GPTQ/AWQ/GGUF are for serving, and the safe merge-then-requantize recipe.

➡️ Next: Module 09 — Synthetic Data Generation

← Module 07: LoRA In Depth