Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT08 — LoRA & QLoRA Duration: 75 minutes Level: Senior Engineer and above Prerequisites: FT00 (The Steering Stack), FT01 (VRAM Math), FT04 (Dataset Formats)
After completing this module, you will be able to:
merge_and_unload() (baked-in) and a standalone adapter (hot-swappable).The adapter that made fine-tuning cheap. Understand the one equation and the rest of the module follows.
LoRA (Low-Rank Adaptation, Hu et al., 2021) rewrites a frozen weight update as a rank-constrained sum of two small matrices:
W = W₀ + ΔW = W₀ + B·A
BA = 0 and the model behaves exactly like the base.The magic is dimensional collapse. A full weight update ΔW lives in a d × k space (for a 4096×4096 layer, that is 16.7M numbers). The LoRA update lives in a d×r + r×k space (at r=8, that is 65K numbers — a 256× reduction). Multiply that across every layer and you get adapters under 1% of the model. The forward pass becomes output = W₀·x + B·A·x; the backward pass updates only A and B.
The crucial initialization detail — B starts at zero — means the adapter is a residual steer. It begins as the identity (the base, unchanged) and learns a perturbation on top. This is why LoRA never destroys the base at step zero, and why it is robust: you cannot make the model worse than the base by accident in the first few steps. Full fine-tuning has no such guarantee.
LoRA is not an arbitrary hack. It rests on the intrinsic dimension hypothesis (Aghajanyan et al., 2020, arXiv:2007.07784): the useful changes during fine-tuning — the ones that actually shift behavior — live in a low-rank subspace of the full parameter space. Aghajanyan showed that you can fine-tune RoBERTa to within 90% of full-FT performance by training only ~0.5% of its parameters, because the effective dimension of the optimization is tiny compared to the nominal parameter count.
This is the same fact the FT00 thesis rests on. Fine-tuning steers behavior; it does not teach knowledge. If you were injecting large amounts of new information, you would need to move large amounts of parameters (high rank, or full FT). But you are not — you are redirecting probability mass the base already has. That redirection is a low-rank operation. Steering is low-rank; that is why a 1% adapter can match full fine-tuning on steering tasks. The Shuttleworth et al. (2024) finding that LoRA and full-FT produce structurally different weight matrices (different geometry, similar behavior) is exactly what you would expect if the task is low-rank steering rather than full-rank knowledge insertion.
For a layer of size d × k with rank r, LoRA adds (d·r + r·k) = r·(d+k) trainable parameters. For a typical 7B model with all linear layers targeted at r=8: ~40M trainable params out of 7B — about 0.5%. At r=16: ~1%. At r=64: ~2%. These numbers are why the optimizer states (the FT01 killer at 16 bytes/param) stay small: they are only computed for A and B, never for W₀.
Every LoRA adapter is specified by four parameters. Get them right and the adapter works; get them wrong and you underfit, overfit, or pay full-FT memory for LoRA quality.
The rank is how many independent directions of change the adapter can express. Defaults:
The rule of thumb: start at r=16, escalate only with evidence. Too-low rank underfits (the adapter literally cannot express the needed change). Too-high rank overfits and approaches full-FT memory without full-FT's benefits — at that point you may as well do full FT (Module FT10).
The adapter's contribution is scaled by α/r before being added to W₀: ΔW = (α/r)·B·A. The convention — and it is a strong convention, not a law — is α ≈ 2×r (so r=8 → α=16; r=16 → α=32). This keeps the effective magnitude of the adapter stable across rank choices, so you do not have to retune the learning rate when you change r.
Why it matters: alpha controls how loudly the adapter speaks over the base. Too-low alpha and the base dominates (underfit — the steer is inaudible). Too-high alpha and the adapter overpowers the base (overfit, catastrophic forgetting). The 2× convention is a well-tested starting point; deviate only with a reason.
Which weight matrices get an adapter. This is the single most impactful choice for adapter quality, and the field's default has shifted dramatically since the original LoRA paper.
["q_proj", "v_proj"]. Cheap, weak. This was the default when VRAM was scarce and people were cautious.["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]. Stronger, more reliable, the recommended setting. Targeting the MLP layers (gate_proj, up_proj, down_proj) is what closes most of the remaining quality gap to full FT.The reason all-linear works better: behavior steering does not live only in attention. The MLP layers store and transform features; steering a persona or a format touches the feed-forward pathway too. Empirically, targeting all linear modules at moderate rank matches or beats attention-only at high rank — for similar trainable-param counts, because you distribute the capacity more sensibly. The 2024 modern default is all-linear. Use it unless you have a specific reason not to.
Regularization on the adapter inputs. Typically 0.05–0.1 for SFT, sometimes 0 for very small datasets where you want every signal. The original LoRA paper used 0.05 on attention. Dropout matters more for higher ranks and smaller datasets (overfit risk); for a standard r=16 SFT on a few thousand examples, 0.05 is the safe default.
from peft import LoraConfig
lora_config = LoraConfig(
r=16,
lora_alpha=32, # convention: alpha ≈ 2×r
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj", # attention
"gate_proj", "up_proj", "down_proj", # MLP (modern default)
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
That is a complete, production-quality LoRA config. Memorize it as the starting point.
LoRA made adapters cheap. QLoRA (Dettmers et al., 2023, arXiv:2305.14314) made the base cheap — by freezing it at 4-bit. Three innovations, each necessary.
Plain LoRA keeps the base at FP16/BF16. For a 7B model that is ~14 GB of weights alone, before any optimizer states or activations (FT01). That fits an A100 but not a consumer 24GB card once you add training overhead. QLoRA's insight: freeze the base at 4-bit and train LoRA adapters on top of it in 16-bit. The base shrinks 4× (14 GB → ~3.5 GB), the adapters stay full-precision, and the whole job fits where it could not before. Three innovations make this work without quality loss.
Standard 4-bit quantization divides the value range into 16 equal bins (uniform). But neural-network weights are not uniformly distributed — they are approximately Gaussian (bell-shaped), with most weights near zero and few in the tails. NF4 chooses its 16 quantile bins to match the normal distribution: more bins near zero where the weights cluster, fewer in the tails. This is information-theoretically optimal for normally-distributed weights — it minimizes the expected quantization error for that distribution. The result: 4-bit storage with quality closer to what you would expect from 6-8 bit uniform quantization.
When you quantize a tensor to 4-bit, you store scaling constants (one per block of ~64 weights) in FP32 to dequantize on the fly. For a 7B model, those constants add up — roughly 0.37 bits per parameter of overhead, or ~0.3 GB on a 7B model. Double quantization quantizes the quantization constants themselves (the scaling factors) to 8-bit. That saves ~0.3 GB — small in absolute terms, but on a 24GB card fighting for every gigabyte, it is the difference between fitting and OOMing. Small savings, no quality cost, always on.
The sneakiest OOM killer in fine-tuning is not steady-state memory — it is the spike. When the optimizer checkpoints its state to CPU (or to disk), there is a momentary memory spike that can exceed steady-state by several GB. On a card sized to the steady-state budget, this spike triggers OOM and kills the job partway through. QLoRA uses NVIDIA Unified Memory to page optimizer states to CPU memory automatically when these spikes occur, transparently avoiding the OOM. The optimizer state is paged out under memory pressure and paged back in when needed. This is why QLoRA runs reliably on a card that barely fits — paged optimizers absorb the spikes that would otherwise crash the job.
Drop any one and the method degrades. NF4 alone gives you the base footprint but you still pay quantization-constant overhead and risk OOM spikes. Double quantization without NF4 is a minor saving on a worse base quantization. Paged optimizers without the 4-bit base do not save enough. The three compose: NF4 shrinks the base, double quantization trims the overhead, paged optimizers absorb the spikes. Together they are what put 7B fine-tuning on a $1,500 card.
This is FT01 applied to QLoRA. If you have not done FT01, the numbers will feel arbitrary; if you have, they are the same rules of thumb.
For a 7B QLoRA at 4K context:
| Consumer | Bytes/param or rule | 7B footprint |
|---|---|---|
| Base weights (4-bit, NF4) | 0.5 bytes/param | ~3.5 GB |
| Quantization constants (after double quant) | ~0.37 → ~0.18 bits/param | ~0.16 GB |
| LoRA adapter params (r=16, all-linear, BF16) | ~40M × (2 grad + ~8 AdamW) bytes | ~0.4 GB |
| Activations (4K context, batch 1, grad checkpointing, FlashAttn) | rule of thumb | ~4–7 GB |
| CUDA context + fragmentation | fixed overhead | ~1–2 GB |
| Total | ~10–14 GB |
That is why a 7B QLoRA fits on a 24GB RTX 4090 (or a 16GB RTX 4080 with care, or a free Colab T4 for the smaller bases). Compare to the same 7B at LoRA-16-bit (~18–30 GB — needs an A100) or full FT (~100–160 GB — needs multi-A100). The spread between QLoRA and full FT is roughly 10×, and almost no steering task justifies paying it (FT00). This is the economic argument for PEFT, made numerical.
The knobs from FT01 all still apply: turn on gradient_checkpointing=True, use attn_implementation="flash_attention_2", decouple physical from effective batch with gradient accumulation, and use paged_adamw_8bit as the optimizer. The lab builds the full config so you feel each knob.
Five steps. This is the loop you will run for nearly every steering task for the rest of the course.
BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True). The base comes in frozen and quantized.prepare_model_for_kbit_training(model) — this enables gradient checkpointing and prepares the frozen base for adapter attachment. Easy to forget; do not.model = get_peft_model(model, lora_config). This injects the B and A matrices (initialized: A random, B zero) at every target module. Print model.print_trainable_parameters() — you should see <1% trainable.merge_and_unload() vs keep the adapter separateYou have a trained adapter. Now a choice:
merged = model.merge_and_unload(). This computes W₀ := W₀ + (α/r)·B·A for every adapter and produces a single, self-contained model with no adapter runtime overhead. Best for deployment (one artifact, fast inference, can be re-quantized to GGUF/AWQ). The adapter is "baked in."adapter_model.safetensors, often <100 MB). At inference, load the base + PeftModel.from_pretrained(base, adapter_path). Best for hot-swapping — one base, many adapters (a JSON adapter, a pirate-speak adapter, a persona adapter), swap at runtime like loading a config file.The default for production deployment is merge (one artifact, no PEFT dependency at serve time). The default for experimentation and multi-tenant serving is keep separate (swap adapters without reloading the base). Module FT09 covers advanced merging (safetensors stacking, multi-adapter serving); FT19 covers re-quantizing a merged model.
Using ["q_proj", "v_proj"] because the 2021 paper did. On modern bases this leaves significant quality on the table — the MLP layers store features you need to steer. Use all-linear targeting (q/k/v/o_proj + gate/up/down_proj) as the default. Attention-only is the cheap-and-weak setting, not the modern one.
r=2 or r=4 on a task that needs expressivity (multi-behavior steering, complex format). The adapter literally cannot represent the required change; loss plateaus high and behavior never fully shifts. Symptom: the adapter trains but the style never quite lands. Fix: raise r to 16 and compare.
r=128 "to be safe." You overfit the small dataset, approach full-FT memory for LoRA's structural limitations (Shuttleworth: LoRA finds a different solution than full FT, so paying full-FT cost for LoRA geometry is the worst of both). Start at 16; only escalate with evidence.
Alpha divorced from rank (e.g., r=16, α=8 — half the convention). The adapter speaks too quietly and the base dominates; the steer is inaudible. Or α=64 with r=8 (8×) — the adapter overpowers the base and you get catastrophic forgetting. Keep α ≈ 2×r unless you have a reason and a metric.
prepare_model_for_kbit_trainingLoading a 4-bit base and immediately attaching LoRA without preparing it for k-bit training. Gradient checkpointing is off, input embeddings are not made trainable-friendly, and training is unstable or slow. One line, easy to forget, always include it.
Assuming "real" fine-tuning is full FT and QLoRA is a compromise. For steering tasks (FT00), QLoRA is not the compromise — it is the correct tool. Full FT is the exception, justified by specific evidence (Module FT10), not the starting point.
| Term | Definition |
|---|---|
| LoRA | Low-Rank Adaptation; freezes W₀, trains low-rank B·A. Adapter typically <1% of params. |
| W = W₀ + BA | The LoRA equation. W₀ frozen; B (zero-init) and A (random-init) are the trainable pair. |
| Rank (r) | The number of directions of change the adapter can express. Default 16. |
| Alpha (α) | Adapter scaling; applied as α/r. Convention α ≈ 2×r. |
| Target modules | Which weights get adapters. Modern default: all attention + MLP (q/k/v/o + gate/up/down_proj). |
| Intrinsic dimension | Aghajanyan's hypothesis: useful fine-tuning changes live in a low-rank subspace. The license for LoRA. |
| QLoRA | LoRA on a 4-bit-quantized frozen base. 7B in ~10–14 GB. |
| NF4 | NormalFloat 4-bit; quantile bins matched to the normal distribution. Information-theoretically optimal for Gaussian weights. |
| Double quantization | Quantizing the quantization constants themselves to 8-bit; saves ~0.37 bits/param. |
| Paged optimizers | Using NVIDIA Unified Memory to page optimizer states to CPU, avoiding OOM spikes during checkpointing. |
| merge_and_unload() | Bake the adapter into the base, producing one self-contained model for deployment. |
| Hot-swappable adapter | Keep the adapter separate; load base + adapter at runtime to swap behaviors without reloading the base. |
See 07-lab-spec.md — "First Fine-Tune." You will QLoRA-fine-tune a 1–1.5B base on a 500-sample style-steering dataset, merge the adapter, and run inference to confirm the steer took. The full loop in under 30 minutes on a consumer GPU or Colab T4. This is your first real fine-tune.
LoraConfig, get_peft_model, prepare_model_for_kbit_training, merge_and_unload.BitsAndBytesConfig, NF4, double quantization, paged_adamw_8bit.# Module FT08 — LoRA & QLoRA
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT08 — LoRA & QLoRA
**Duration**: 75 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT00 (The Steering Stack), FT01 (VRAM Math), FT04 (Dataset Formats)
---
## Learning Objectives
After completing this module, you will be able to:
1. State the LoRA mechanism — the low-rank decomposition **W = W₀ + BA** where W₀ is frozen and B, A are tiny trainable matrices — and connect it to the "intrinsic dimension" hypothesis (Aghajanyan) and the FT00 thesis that *fine-tuning steers behavior; it does not teach knowledge*.
2. Configure a LoRA adapter from its four knobs — **rank (r), alpha (α), target modules, dropout** — and justify each choice, including the modern default of targeting **all attention + MLP projections** (not attention only).
3. Explain why steering is low-rank and why trainable params often land **under 1%** of the model.
4. Name QLoRA's three innovations (Dettmers et al., 2023) — **NF4 quantization, double quantization, paged optimizers** — and why each is necessary.
5. Derive, from the FT01 VRAM math, why QLoRA unlocks **7B training on a 24GB card** and where the gigabytes actually go.
6. Execute the full QLoRA workflow — load base in 4-bit → attach LoRA adapters → train adapters only → **merge or save** — and decide between `merge_and_unload()` (baked-in) and a standalone adapter (hot-swappable).
---
# 8.1 — The LoRA Mechanism: Steering Is Low-Rank
*The adapter that made fine-tuning cheap. Understand the one equation and the rest of the module follows.*
## The one equation
LoRA (Low-Rank Adaptation, Hu et al., 2021) rewrites a frozen weight update as a rank-constrained sum of two small matrices:
> **W = W₀ + ΔW = W₀ + B·A**
- **W₀** — the pretrained weight matrix (dimensions *d × k*). **Frozen.** No gradient flows into it. This is the base model, untouched.
- **A** — a trainable matrix of shape *r × k*. Initialized with random Gaussian values.
- **B** — a trainable matrix of shape *d × r*. Initialized to **zero**, so at step 0 the adapter contributes *nothing* — `BA = 0` and the model behaves exactly like the base.
- **r** — the **rank**: how many "directions" of change the adapter is allowed to express. Typically 8–64. This is the single most important hyperparameter.
The magic is dimensional collapse. A full weight update ΔW lives in a *d × k* space (for a 4096×4096 layer, that is 16.7M numbers). The LoRA update lives in a *d×r + r×k* space (at r=8, that is 65K numbers — a **256× reduction**). Multiply that across every layer and you get adapters under 1% of the model. The forward pass becomes `output = W₀·x + B·A·x`; the backward pass updates only A and B.
The crucial initialization detail — **B starts at zero** — means the adapter is a *residual* steer. It begins as the identity (the base, unchanged) and learns a perturbation on top. This is why LoRA never destroys the base at step zero, and why it is robust: you cannot make the model *worse* than the base by accident in the first few steps. Full fine-tuning has no such guarantee.
## Why this works: the intrinsic dimension hypothesis
LoRA is not an arbitrary hack. It rests on the **intrinsic dimension** hypothesis (Aghajanyan et al., 2020, arXiv:2007.07784): the *useful* changes during fine-tuning — the ones that actually shift behavior — live in a low-rank subspace of the full parameter space. Aghajanyan showed that you can fine-tune RoBERTa to within 90% of full-FT performance by training only ~0.5% of its parameters, because the *effective* dimension of the optimization is tiny compared to the nominal parameter count.
This is the same fact the FT00 thesis rests on. *Fine-tuning steers behavior; it does not teach knowledge.* If you were injecting large amounts of new information, you would need to move large amounts of parameters (high rank, or full FT). But you are not — you are redirecting probability mass the base already has. That redirection is a low-rank operation. **Steering is low-rank; that is why a 1% adapter can match full fine-tuning on steering tasks.** The Shuttleworth et al. (2024) finding that LoRA and full-FT produce *structurally different* weight matrices (different geometry, similar behavior) is exactly what you would expect if the task is low-rank steering rather than full-rank knowledge insertion.
### The trainable-params arithmetic
For a layer of size *d × k* with rank *r*, LoRA adds *(d·r + r·k) = r·(d+k)* trainable parameters. For a typical 7B model with all linear layers targeted at r=8: ~40M trainable params out of 7B — about **0.5%**. At r=16: ~1%. At r=64: ~2%. These numbers are why the optimizer states (the FT01 killer at 16 bytes/param) stay small: they are only computed for A and B, never for W₀.
---
# 8.2 — The LoRA Config: Four Knobs
*Every LoRA adapter is specified by four parameters. Get them right and the adapter works; get them wrong and you underfit, overfit, or pay full-FT memory for LoRA quality.*
## Knob 1 — Rank (r)
The rank is how many independent directions of change the adapter can express. Defaults:
- **r = 8** — the historical default and still fine for narrow tasks (single-style format steering, persona).
- **r = 16–32** — the modern default for most SFT. A good balance of capacity and regularization.
- **r = 64+** — for complex multi-task or knowledge-adjacent steering. Watch VRAM and overfitting.
The rule of thumb: **start at r=16, escalate only with evidence.** Too-low rank underfits (the adapter literally cannot express the needed change). Too-high rank overfits and approaches full-FT memory without full-FT's benefits — at that point you may as well do full FT (Module FT10).
## Knob 2 — Alpha (α) — the scaling
The adapter's contribution is scaled by `α/r` before being added to W₀: `ΔW = (α/r)·B·A`. The **convention** — and it is a strong convention, not a law — is **α ≈ 2×r** (so r=8 → α=16; r=16 → α=32). This keeps the effective magnitude of the adapter stable across rank choices, so you do not have to retune the learning rate when you change r.
Why it matters: alpha controls *how loudly the adapter speaks over the base*. Too-low alpha and the base dominates (underfit — the steer is inaudible). Too-high alpha and the adapter overpowers the base (overfit, catastrophic forgetting). The 2× convention is a well-tested starting point; deviate only with a reason.
## Knob 3 — Target modules (the biggest quality lever)
Which weight matrices get an adapter. This is the single most impactful choice for adapter quality, and the field's default has shifted dramatically since the original LoRA paper.
- **Original LoRA (2021): attention only** — typically `["q_proj", "v_proj"]`. Cheap, weak. This was the default when VRAM was scarce and people were cautious.
- **Modern default (2024+): all attention + all MLP projections** — `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`. Stronger, more reliable, the recommended setting. Targeting the MLP layers (`gate_proj`, `up_proj`, `down_proj`) is what closes most of the remaining quality gap to full FT.
The reason all-linear works better: behavior steering does not live only in attention. The MLP layers store and transform features; steering a persona or a format touches the feed-forward pathway too. Empirically, targeting all linear modules at moderate rank matches or beats attention-only at high rank — for similar trainable-param counts, because you distribute the capacity more sensibly. **The 2024 modern default is all-linear. Use it unless you have a specific reason not to.**
## Knob 4 — Dropout
Regularization on the adapter inputs. Typically **0.05–0.1** for SFT, sometimes 0 for very small datasets where you want every signal. The original LoRA paper used 0.05 on attention. Dropout matters more for higher ranks and smaller datasets (overfit risk); for a standard r=16 SFT on a few thousand examples, 0.05 is the safe default.
### The PEFT config, made concrete
```python
from peft import LoraConfig
lora_config = LoraConfig(
r=16,
lora_alpha=32, # convention: alpha ≈ 2×r
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj", # attention
"gate_proj", "up_proj", "down_proj", # MLP (modern default)
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
```
That is a complete, production-quality LoRA config. Memorize it as the starting point.
---
# 8.3 — QLoRA: Three Innovations That Fit 7B on a 24GB Card
*LoRA made adapters cheap. QLoRA (Dettmers et al., 2023, arXiv:2305.14314) made the **base** cheap — by freezing it at 4-bit. Three innovations, each necessary.*
## The problem QLoRA solves
Plain LoRA keeps the base at FP16/BF16. For a 7B model that is ~14 GB of weights alone, before any optimizer states or activations (FT01). That fits an A100 but not a consumer 24GB card once you add training overhead. QLoRA's insight: **freeze the base at 4-bit and train LoRA adapters on top of it in 16-bit.** The base shrinks 4× (14 GB → ~3.5 GB), the adapters stay full-precision, and the whole job fits where it could not before. Three innovations make this work without quality loss.
## Innovation 1 — NF4 (NormalFloat 4-bit) quantization
Standard 4-bit quantization divides the value range into 16 equal bins (uniform). But neural-network weights are **not uniformly distributed** — they are approximately Gaussian (bell-shaped), with most weights near zero and few in the tails. NF4 chooses its 16 quantile bins to match the normal distribution: more bins near zero where the weights cluster, fewer in the tails. This is **information-theoretically optimal** for normally-distributed weights — it minimizes the expected quantization error for that distribution. The result: 4-bit storage with quality closer to what you would expect from 6-8 bit uniform quantization.
## Innovation 2 — Double quantization
When you quantize a tensor to 4-bit, you store scaling constants (one per block of ~64 weights) in FP32 to dequantize on the fly. For a 7B model, those constants add up — roughly **0.37 bits per parameter** of overhead, or ~0.3 GB on a 7B model. Double quantization **quantizes the quantization constants themselves** (the scaling factors) to 8-bit. That saves ~0.3 GB — small in absolute terms, but on a 24GB card fighting for every gigabyte, it is the difference between fitting and OOMing. Small savings, no quality cost, always on.
## Innovation 3 — Paged optimizers
The sneakiest OOM killer in fine-tuning is not steady-state memory — it is the **spike**. When the optimizer checkpoints its state to CPU (or to disk), there is a momentary memory spike that can exceed steady-state by several GB. On a card sized to the steady-state budget, this spike triggers OOM and kills the job partway through. QLoRA uses **NVIDIA Unified Memory** to page optimizer states to CPU memory automatically when these spikes occur, transparently avoiding the OOM. The optimizer state is paged out under memory pressure and paged back in when needed. This is why QLoRA runs reliably on a card that *barely* fits — paged optimizers absorb the spikes that would otherwise crash the job.
### Why all three are necessary
Drop any one and the method degrades. NF4 alone gives you the base footprint but you still pay quantization-constant overhead and risk OOM spikes. Double quantization without NF4 is a minor saving on a worse base quantization. Paged optimizers without the 4-bit base do not save enough. The three compose: NF4 shrinks the base, double quantization trims the overhead, paged optimizers absorb the spikes. Together they are what put 7B fine-tuning on a $1,500 card.
---
# 8.4 — The VRAM Math, Made Concrete
*This is FT01 applied to QLoRA. If you have not done FT01, the numbers will feel arbitrary; if you have, they are the same rules of thumb.*
For a **7B QLoRA at 4K context**:
| Consumer | Bytes/param or rule | 7B footprint |
| --- | --- | --- |
| Base weights (4-bit, NF4) | 0.5 bytes/param | ~3.5 GB |
| Quantization constants (after double quant) | ~0.37 → ~0.18 bits/param | ~0.16 GB |
| LoRA adapter params (r=16, all-linear, BF16) | ~40M × (2 grad + ~8 AdamW) bytes | ~0.4 GB |
| Activations (4K context, batch 1, grad checkpointing, FlashAttn) | rule of thumb | ~4–7 GB |
| CUDA context + fragmentation | fixed overhead | ~1–2 GB |
| **Total** | | **~10–14 GB** |
That is why a 7B QLoRA fits on a **24GB RTX 4090** (or a 16GB RTX 4080 with care, or a free Colab T4 for the smaller bases). Compare to the same 7B at LoRA-16-bit (~18–30 GB — needs an A100) or full FT (~100–160 GB — needs multi-A100). The spread between QLoRA and full FT is roughly **10×**, and almost no steering task justifies paying it (FT00). This is the economic argument for PEFT, made numerical.
The knobs from FT01 all still apply: turn on `gradient_checkpointing=True`, use `attn_implementation="flash_attention_2"`, decouple physical from effective batch with gradient accumulation, and use `paged_adamw_8bit` as the optimizer. The lab builds the full config so you feel each knob.
---
# 8.5 — The Full QLoRA Workflow
*Five steps. This is the loop you will run for nearly every steering task for the rest of the course.*
1. **Load the base in 4-bit.** Use `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)`. The base comes in frozen and quantized.
2. **Prepare the model for k-bit training.** `prepare_model_for_kbit_training(model)` — this enables gradient checkpointing and prepares the frozen base for adapter attachment. Easy to forget; do not.
3. **Attach the LoRA adapters.** `model = get_peft_model(model, lora_config)`. This injects the B and A matrices (initialized: A random, B zero) at every target module. Print `model.print_trainable_parameters()` — you should see **<1% trainable**.
4. **Train the adapters only.** The optimizer only touches A and B; the 4-bit base never moves. Standard SFTTrainer (TRL) or a manual loop.
5. **Merge or save.** Two deployment options (next section).
### Merging: `merge_and_unload()` vs keep the adapter separate
You have a trained adapter. Now a choice:
- **Merge into the base** — `merged = model.merge_and_unload()`. This computes `W₀ := W₀ + (α/r)·B·A` for every adapter and produces a single, self-contained model with no adapter runtime overhead. Best for **deployment** (one artifact, fast inference, can be re-quantized to GGUF/AWQ). The adapter is "baked in."
- **Keep the adapter separate** — save just the adapter weights (`adapter_model.safetensors`, often <100 MB). At inference, load the base + `PeftModel.from_pretrained(base, adapter_path)`. Best for **hot-swapping** — one base, many adapters (a JSON adapter, a pirate-speak adapter, a persona adapter), swap at runtime like loading a config file.
The default for production deployment is **merge** (one artifact, no PEFT dependency at serve time). The default for experimentation and multi-tenant serving is **keep separate** (swap adapters without reloading the base). Module FT09 covers advanced merging (safetensors stacking, multi-adapter serving); FT19 covers re-quantizing a merged model.
---
## Anti-Patterns
### Attention-only targeting on a modern model
Using `["q_proj", "v_proj"]` because the 2021 paper did. On modern bases this leaves significant quality on the table — the MLP layers store features you need to steer. **Use all-linear targeting** (`q/k/v/o_proj` + `gate/up/down_proj`) as the default. Attention-only is the cheap-and-weak setting, not the modern one.
### Too-low rank (underfit)
r=2 or r=4 on a task that needs expressivity (multi-behavior steering, complex format). The adapter literally cannot represent the required change; loss plateaus high and behavior never fully shifts. Symptom: the adapter trains but the style never quite lands. Fix: raise r to 16 and compare.
### Too-high rank (overfit + memory)
r=128 "to be safe." You overfit the small dataset, approach full-FT memory for LoRA's structural limitations (Shuttleworth: LoRA finds a different solution than full FT, so paying full-FT cost for LoRA geometry is the worst of both). Start at 16; only escalate with evidence.
### Alpha poorly tuned
Alpha divorced from rank (e.g., r=16, α=8 — half the convention). The adapter speaks too quietly and the base dominates; the steer is inaudible. Or α=64 with r=8 (8×) — the adapter overpowers the base and you get catastrophic forgetting. **Keep α ≈ 2×r** unless you have a reason and a metric.
### Forgetting `prepare_model_for_kbit_training`
Loading a 4-bit base and immediately attaching LoRA without preparing it for k-bit training. Gradient checkpointing is off, input embeddings are not made trainable-friendly, and training is unstable or slow. One line, easy to forget, always include it.
### Treating QLoRA as inferior by default
Assuming "real" fine-tuning is full FT and QLoRA is a compromise. For steering tasks (FT00), QLoRA is not the compromise — it is the correct tool. Full FT is the exception, justified by specific evidence (Module FT10), not the starting point.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **LoRA** | Low-Rank Adaptation; freezes W₀, trains low-rank B·A. Adapter typically <1% of params. |
| **W = W₀ + BA** | The LoRA equation. W₀ frozen; B (zero-init) and A (random-init) are the trainable pair. |
| **Rank (r)** | The number of directions of change the adapter can express. Default 16. |
| **Alpha (α)** | Adapter scaling; applied as α/r. Convention α ≈ 2×r. |
| **Target modules** | Which weights get adapters. Modern default: all attention + MLP (q/k/v/o + gate/up/down_proj). |
| **Intrinsic dimension** | Aghajanyan's hypothesis: useful fine-tuning changes live in a low-rank subspace. The license for LoRA. |
| **QLoRA** | LoRA on a 4-bit-quantized frozen base. 7B in ~10–14 GB. |
| **NF4** | NormalFloat 4-bit; quantile bins matched to the normal distribution. Information-theoretically optimal for Gaussian weights. |
| **Double quantization** | Quantizing the quantization constants themselves to 8-bit; saves ~0.37 bits/param. |
| **Paged optimizers** | Using NVIDIA Unified Memory to page optimizer states to CPU, avoiding OOM spikes during checkpointing. |
| **merge_and_unload()** | Bake the adapter into the base, producing one self-contained model for deployment. |
| **Hot-swappable adapter** | Keep the adapter separate; load base + adapter at runtime to swap behaviors without reloading the base. |
---
## Lab Exercise
See `07-lab-spec.md` — "First Fine-Tune." You will QLoRA-fine-tune a 1–1.5B base on a 500-sample style-steering dataset, merge the adapter, and run inference to confirm the steer took. The full loop in under 30 minutes on a consumer GPU or Colab T4. This is your first real fine-tune.
---
## References
1. **Hu et al. (2021)** — *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685. The adapter method; the W = W₀ + BA decomposition.
2. **Dettmers et al. (2023)** — *QLoRA: Efficient Finetuning of Quantized LLMs*. arXiv:2305.14314. NF4, double quantization, paged optimizers. 7B on a consumer GPU.
3. **Aghajanyan et al. (2020)** — *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. arXiv:2007.07784. The theoretical license for low-rank steering.
4. **Shuttleworth et al. (2024)** — *An Illusion of Equivalence: LoRA vs Full Fine-Tuning*. arXiv:2410.21228. LoRA and full FT produce structurally different weights.
5. **HuggingFace PEFT docs** — `LoraConfig`, `get_peft_model`, `prepare_model_for_kbit_training`, `merge_and_unload`.
6. **bitsandbytes docs** — `BitsAndBytesConfig`, NF4, double quantization, `paged_adamw_8bit`.
7. **FT00 — The Steering Stack** — the thesis this module operationalizes (steering is low-rank).
8. **FT01 — VRAM Math** — the three consumers and the rules of thumb that make the QLoRA footprint derivable, not memorized.