Module 07 — LoRA In Depth

Goal: Truly understand LoRA — the most important fine-tuning technique in the field — including the (gentle) math, every hyperparameter, where to apply it, and how to run it in code. By the end you’ll be able to configure a LoRA fine-tune confidently and explain why each setting matters, which is exactly what makes someone an expert rather than a copy-paster.

1. The big idea in one breath

LoRA freezes the model’s giant weight matrices and learns a small, low-rank “patch” for each one. The patch is the product of two skinny matrices, so it has very few parameters — yet it can be added back into the original weights for free, with zero extra cost when you actually use the model.

If you understood Module 06’s “add a small rudder to a ship that already knows how to sail,” LoRA is the best-engineered rudder. Now let’s see exactly how it’s built.

2. The intuition behind “low-rank”

Recall from Module 01 that a model’s behavior lives in big weight matrices (the Q, K, V, O projections in attention, and the MLP layers). When you fully fine-tune, you change one of these matrices W into W + ΔW, where ΔW is the update — “how much each weight moved.”

The key empirical discovery behind LoRA (from the 2021 LoRA paper, building on the idea that adaptation is low-dimensional, Module 06 §2) is this:

That update ΔW — though it’s a big matrix — doesn’t need to be full of independent information. It can be well-approximated by something much “simpler”: a low-rank matrix.

What does “low-rank” mean, intuitively? A matrix’s rank is, loosely, “how much genuinely independent information it contains.” A big grid of numbers that’s actually just a few patterns repeated and scaled has low rank — you can reconstruct the whole thing from a little. LoRA bets that the change a model needs for your task is exactly this kind of “a little goes a long way” object.

And here’s the beautiful mathematical fact that makes it cheap: any low-rank matrix can be written as the product of two skinny matrices. If ΔW is a 4096×4096 grid (≈16.7 million numbers) but has rank 8, you can build it from:

a tall-skinny matrix A of size 4096×8, and
a short-wide matrix B of size 8×4096,

because B × A produces a 4096×4096 matrix — but A and B together hold only 4096×8 + 8×4096 ≈ 65,000 numbers instead of 16.7 million. That’s a ~250× reduction in what you have to train, for this one matrix. Across the whole model, that’s why LoRA trains well under 1% of the parameters.

3. The mechanism, drawn

For each weight matrix LoRA targets, it does this:

        original (FROZEN, not trained)
                 W
                 │
   input  ───────┼───────────────►  W·x        ┐
     x           │                              │  add them
                 │   ┌──────┐   ┌──────┐        │  together
                 └──►│  A   │──►│  B   │──►  B·A·x ┘  ──►  output
                     └──────┘   └──────┘
                   (trainable) (trainable)
                   tall-skinny  short-wide
                   rank = r      rank = r

   Effective new weight:  W + (B·A) · scaling
   We train ONLY A and B.  W never moves.

In words:

The original weight W stays frozen — it still does its job on the input x, producing W·x.
In parallel, the input also flows through the two small trainable matrices: first A, then B, producing B·A·x.
The two results are added. The model’s new behavior is W·x + (scaling)·B·A·x.
During training, gradients only flow to A and B. W is untouched — so you don’t pay the memory cost of its gradients or optimizer state (Module 03 §3, §6). This is the entire memory win.

Two small but important initialization details (so training starts sanely):

A is initialized to small random values.
B is initialized to zero.

Because B starts at zero, B·A starts at zero, so at step 0 the LoRA patch contributes nothing — the model behaves exactly like the original base model, and learning starts from that safe point and gently moves away. Elegant, and worth remembering: a freshly-initialized LoRA = the untouched base model.

4. The “merge” trick: zero inference cost

Here’s LoRA’s killer feature versus adapters (Module 06 §4). After training, you can compute B·A·scaling once and add it directly into W:

   W_merged = W + (scaling)·B·A

Now W_merged is just a normal weight matrix of the same size and shape as the original. You can drop it back into the model and run inference with exactly the same speed and memory as the base model — the extra A/B path is gone, folded in. This is why people say LoRA has no inference overhead: you only pay during training; at deployment you can merge and forget it existed.

You don’t have to merge — you can also keep the adapter separate and apply it on the fly, which is what enables multi-LoRA serving (one base model + many swappable adapters, Module 14). Merging vs keeping-separate is a real deployment choice, and now you understand both sides.

5. The hyperparameters of LoRA (this is the heart of the module)

Configuring LoRA comes down to a handful of settings. Understand each one and you can tune LoRA for anything.

5a. `r` — the rank (the most important LoRA knob)

r is the rank of the patch — the “8” in our example, the width of the bottleneck between A and B. It controls how much capacity the adapter has:

Low r (e.g., 4–8): fewer parameters, less capacity, trains fast, resists overfitting, smaller files. Often plenty for style/format/narrow-task tuning.
High r (e.g., 32, 64, 128): more capacity to learn complex or large changes, but more parameters, more memory, and more overfitting risk on small data.

Practical guidance: start at r = 8 or 16. Increase only if evaluation shows the model can’t capture the behavior (underfitting). More rank is not automatically better — beyond what the task needs, you just add overfitting risk and cost. This “start small, raise if needed” discipline is a mark of experience.

5b. `lora_alpha` — the scaling factor

lora_alpha sets the scaling in W + scaling·B·A. The actual scaling applied is alpha / r. Think of it as how strongly the LoRA patch is allowed to influence the model relative to its rank.

The widely used heuristic: set alpha = r (scaling = 1) or alpha = 2·r (scaling = 2). Many recipes use r=16, alpha=32.
Intuition: alpha is a bit like a per-adapter learning-rate multiplier. Too high and the patch dominates and destabilizes; too low and it barely matters.
A subtlety worth knowing: because scaling is alpha/r, if you change r you may want to change alpha to keep the effective influence steady. (A variant called rsLoRA rescales differently to make high ranks behave better — §10.)

Beginner-friendly rule: set r = 16, alpha = 32, and move on. Tune later only if needed.

5c. `target_modules` — which weight matrices get a LoRA patch

You decide which of the model’s matrices receive an adapter. Recall the candidates from Module 01: the attention projections q_proj, k_proj, v_proj, o_proj and the MLP projections (often gate_proj, up_proj, down_proj).

The original LoRA paper applied it mainly to the attention matrices (q_proj, v_proj).
Modern practice often targets all linear layers (attention and MLP) for stronger results — many libraries support target_modules="all-linear" to do this automatically. More targets = more capacity and memory, but usually better quality.
Practical default: target all linear layers if memory allows; otherwise at least q_proj, k_proj, v_proj, o_proj. Getting the names right for your specific model matters — they vary by architecture, and a typo means that layer silently gets no adapter. (Inspect the model’s module names if unsure.)

5d. `lora_dropout`

Standard dropout (Module 03 §7 regularization) applied within the LoRA path to reduce overfitting. A small value like 0.05 is common. Use a bit more if you’re overfitting, zero if you have lots of data.

5e. `bias`

Whether to also train bias terms. Usually left as "none" (don’t). Minor knob; default is fine.

Putting the LoRA config together

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,                       # rank — capacity of the patch (§5a)
    lora_alpha=32,              # scaling = alpha/r = 2 (§5b)
    target_modules="all-linear",# which matrices get patched (§5c)
    lora_dropout=0.05,          # regularization (§5d)
    bias="none",                # don't train biases (§5e)
    task_type="CAUSAL_LM",      # we're doing language modeling
)

You can now read any LoRA config in the wild and know what every line does.

6. LoRA in code: turning the Module 05 SFT script into LoRA

The payoff: making the full-fine-tuning SFT script from Module 05 into a LoRA fine-tune is almost no extra work. You add a LoraConfig and pass it to the trainer. Here’s the diff, conceptually:

# ... same imports, tokenizer, dataset loading as Module 05 ...
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto")

lora_config = LoraConfig(          # ← NEW: the only conceptual addition
    r=16, lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
)

config = SFTConfig(
    output_dir="./lora-out",
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,            # ← NOTE: ~10x higher than full FT (Module 05 §5)
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True, logging_steps=10,
    eval_strategy="steps", eval_steps=50,
    max_seq_length=2048, packing=True,
)

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    processing_class=tokenizer,
    peft_config=lora_config,       # ← NEW: pass the LoRA config; trainer freezes
                                   #    the base and adds A/B matrices for you
)

trainer.train()
trainer.save_model("./lora-adapter")   # saves a TINY adapter (a few MB), not the whole model

Three things to notice, all consequences of what you now understand:

Learning rate is ~10× higher than full fine-tuning (2e-4 vs 2e-5). LoRA trains fresh small matrices that start at zero, so they tolerate (and need) bigger steps (Module 03 §4, Module 05 §5).
save_model saves a tiny adapter, not a full model copy — the portable artifact superpower from Module 06 §8.
Everything else is identical to SFT. LoRA didn’t change what you’re teaching (your data, the loss, masking) — only how cheaply you teach it. That’s the whole point of Module 05’s claim that “LoRA is just a cheaper way to do SFT.”

7. Using your trained LoRA adapter

After training you have a small adapter folder. Two ways to use it:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto")
tok  = AutoTokenizer.from_pretrained(model_name)

# Option A: load base + attach adapter (keeps them separate; good for swapping)
model = PeftModel.from_pretrained(base, "./lora-adapter")

# Option B: merge the adapter into the base for zero-overhead inference (§4)
model = PeftModel.from_pretrained(base, "./lora-adapter")
model = model.merge_and_unload()          # now it's a normal model; A/B folded into W
model.save_pretrained("./merged-model")   # a full standalone model you can deploy/serve

Choose Option A when you want to serve many adapters off one base (Module 14) or keep iterating. Choose Option B (merge) when you’re deploying a single specialized model and want maximum inference speed. You now understand the trade-off behind a choice that confuses most beginners.

8. Memory and quality: what LoRA actually buys you

Concrete sense of the win (numbers are illustrative, to build intuition):

Trainable parameters: full FT of a 7B model trains ~7,000,000,000 params; LoRA at r=16 on attention+MLP trains on the order of a few to tens of millions — well under 1%.
Memory: because you skip gradients and optimizer state for the frozen 99%+, peak memory drops dramatically — turning a multi-GPU job into a single-GPU one. (QLoRA, next module, pushes this even further by compressing the frozen base itself.)
Quality: for most specialization tasks, well-tuned LoRA reaches within a hair of full fine-tuning — often indistinguishable on the actual task. It tends to underperform full FT only when you’re trying to teach very large, broad new capabilities, where more capacity genuinely helps.
Forgetting & overfitting: generally better than full FT (frozen base + few params), per Module 05 §7 and Module 03 §7.

The honest summary: LoRA gives you ~95–100% of the result for a tiny fraction of the cost, with bonus robustness. That trade is so good it became the default.

9. LoRA pitfalls and how experts avoid them

The specific ways LoRA goes wrong — learn these and you’ll debug faster than most:

Wrong/typo’d target_modules. The named layer silently gets no adapter → weak or no learning. Verify module names against your model architecture.
Rank too low for the task → underfitting. Behavior won’t “stick.” Raise r (and re-check alpha).
Rank too high on small data → overfitting and wasted memory. Lower r, add lora_dropout, fewer epochs.
Forgetting to use the right learning rate. Using a full-FT LR (2e-5) with LoRA often underfits (too timid); use the LoRA range (1e-4–3e-4).
Mismatched precision when merging. Merging a LoRA trained in one precision into a base loaded in another can subtly degrade quality; merge carefully and evaluate the merged model, don’t assume.
Template/masking/data problems. Reminder: most “LoRA didn’t work” cases are actually data or chat-template problems (Module 04, Module 05 §8), not LoRA settings. Check those first.
Evaluating only by vibes. Always measure (Module 11). LoRA changes can be subtle; eyeballing five outputs lies to you.

10. LoRA variants (know they exist; reach for them later)

Once you’ve mastered plain LoRA, these refinements chase extra quality or efficiency. You don’t need them to be productive — but recognizing the names marks fluency:

QLoRA — LoRA on a 4-bit quantized frozen base, for fitting huge models on one GPU. So important it gets its own module (08).
DoRA (Weight-Decomposed LoRA) — splits each weight into “magnitude” and “direction” and adapts them separately; often a small quality bump over LoRA at similar cost.
rsLoRA (rank-stabilized LoRA) — changes the scaling so that high ranks train more stably; useful when you genuinely need large r.
AdaLoRA — lets the method allocate rank adaptively across layers (more where it helps, less where it doesn’t).
PiSSA — a smarter initialization of A/B (from the base weights’ principal components) that can converge faster/better.
LoHa / LoKr — alternative low-rank factorizations (Hadamard/Kronecker) used especially in image models.
VeRA, IA³, and others — even more parameter-frugal cousins.

The meta-lesson: they’re all variations on the same core idea you now own — represent the task-specific update compactly and train only that. Understanding plain LoRA deeply means you can pick up any of these in an afternoon, exactly the durable understanding Module 00 promised.

Module 07 checklist

I can explain LoRA in one sentence and why low-rank updates are enough.
I can describe the frozen-W + B·A mechanism and why B starts at zero.
I can explain the merge trick and why LoRA has no inference overhead.
I can define r, lora_alpha, target_modules, lora_dropout and give sensible defaults.
I can turn an SFT script into a LoRA script and know why the LR is higher.
I can load/merge an adapter and choose between separate vs merged for deployment.
I can list LoRA’s common pitfalls and name a few variants.

➡️ Next: Module 08 — QLoRA & Quantization