Diagrams — Module FT08: LoRA & QLoRA

Module: FT08 — LoRA & QLoRA Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The LoRA Decomposition: W = W₀ + BA

Type: Mechanism / equation visualization Purpose: The one equation the module rests on. W₀ is frozen; B (zero-init) and A (random-init) are tiny trainable matrices that compose into a low-rank update ΔW = BA. Reading the diagram: The frozen pretrained W₀ (large, d×k) stays put. A (r×k) and B (d×r) are injected on a side path. Their product BA is added to W₀. B starts at zero so ΔW = 0 at step 0 (identity with the base).

flowchart LR
  X["input x\n(d-vector)"]
  X --> W0["W₀\nFROZEN\npretrained\n(d × k)\n~16.7M params\n(for 4096²)"]
  X --> A["A\nTRAINABLE\nrandom-init\n(r × k)\nr=8 → tiny"]
  A --> B["B\nTRAINABLE\nZERO-init\n(d × r)\nstart = 0"]
  B --> Scale["× α/r\n(scaling)"]
  Scale --> BA["ΔW = BA\nlow-rank update"]
  W0 --> Sum(("+"))
  BA --> Sum
  Sum --> Out["output\n= W₀x + (α/r)BAx\nmodel starts == base"]

  B -.->|"step 0: B=0\n→ ΔW=0\n→ adapter is identity"| Note0["robust: cannot break\nthe base on step 1"]
  BA -.->|"256× fewer params\nthan full ΔW"| Note1["this is why\n<1% trainable"]

  style W0 fill:#14141f,stroke:rgba(148,148,160,0.5),stroke-width:1.5px,color:#9494a0
  style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style B fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style Scale fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style BA fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style Sum fill:#08080c,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style X fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Note0 fill:#08080c,stroke:rgba(130,224,170,0.4),color:#82e0aa
  style Note1 fill:#08080c,stroke:rgba(130,224,170,0.4),color:#82e0aa

Diagram 2 — The LoRA Config Parameter Map

Type: Config-knob reference Purpose: The four knobs that define every LoRA adapter, with the modern default for each. Reading the diagram: Four knobs (rank, alpha, target modules, dropout). The center node is the resulting LoraConfig. Defaults shown are the modern production starting point (r=16, α=32, all-linear, dropout 0.05).

flowchart TD
  Config["LoraConfig\nthe adapter definition"]

  R["RANK (r)\n# directions of change\nDEFAULT: 16\nlow (8): narrow tasks\nhigh (64): multi-task, watch overfit"]
  Alpha["ALPHA (α)\nscaling: applied as α/r\nCONVENTION: α ≈ 2×r\n(r=16 → α=32)\ntoo low → base dominates\ntoo high → forgetting"]
  Tgt["TARGET MODULES\nbiggest quality lever\nDEFAULT: ALL linear\nq,k,v,o_proj + gate,up,down_proj\nNOT attention-only (2021 default)"]
  Drop["DROPOUT\nregularization on adapter\nDEFAULT: 0.05\nsmall data / high r → raise\nlarge data → 0 ok"]

  R --> Config
  Alpha --> Config
  Tgt --> Config
  Drop --> Config

  Config --> PEFT["get_peft_model(model, config)\n→ <1% params trainable\n→ B zero-init: starts == base"]

  style Config fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#e4e4e8
  style R fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Alpha fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Tgt fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Drop fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style PEFT fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa

Diagram 3 — QLoRA's Three Innovations

Type: Component decomposition Purpose: Why QLoRA works — three innovations, each necessary, that compose to fit 7B on a 24GB card. Reading the diagram: The frozen base is quantized three ways down. NF4 shrinks the weights to 4-bit optimal-for-Gaussian. Double quant trims the constants. Paged optimizers absorb the checkpointing spikes. Drop any one and the method degrades.

flowchart TD
  Base["7B base weights\nFP16: ~14 GB\n(does NOT fit a 24GB card\nonce you add training overhead)"]

  Base --> NF4["INNOVATION 1 — NF4\nNormalFloat 4-bit\nquantile bins matched to\nthe NORMAL distribution\n(info-theoretically optimal\nfor Gaussian weights)\n→ 14 GB ≈ 3.5 GB"]
  NF4 --> DQ["INNOVATION 2 — Double Quantization\nquantize the QUANTIZATION CONSTANTS\nthemselves to 8-bit\nsaves ~0.37 bits/param\n→ trims ~0.3 GB overhead"]
  DQ --> Paged["INNOVATION 3 — Paged Optimizers\nNVIDIA Unified Memory pages\noptimizer state to CPU on spikes\n(avoids OOM during checkpointing)\n→ absorbs the killer spikes"]

  NF4 -.without.-> L1["base worse quality\n(uniform 4-bit bins)"]
  DQ -.without.-> L2["~0.3 GB overhead\n→ may OOM at margin"]
  Paged -.without.-> L3["steady-state fits\nbut spikes crash the job"]

  Paged --> Train["LoRA adapters train in BF16\non the 4-bit frozen base\n→ 7B QLoRA: ~10–14 GB\n→ fits RTX 4090 (24GB)"]

  style Base fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style NF4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style DQ fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style Paged fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style L1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style L2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style L3 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style Train fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa

Diagram 4 — The Full QLoRA Workflow

Type: Sequential workflow Purpose: The five-step loop you run for nearly every steering task. Load 4-bit → prepare → attach → train → merge (or save). Reading the diagram: Left to right. Each node is a concrete API call. The branch at the end is the merge-vs-keep-adapter decision.

flowchart LR
  L["1. LOAD base in 4-bit\nBitsAndBytesConfig(\n  load_in_4bit=True,\n  bnb_4bit_quant_type='nf4',\n  bnb_4bit_use_double_quant=True,\n  bnb_4bit_compute_dtype=bf16)"]
  P["2. PREPARE k-bit\nprepare_model_for_\nkbit_training(model)\n(grad checkpointing, etc.)\n—the step everyone forgets"]
  A["3. ATTACH adapters\nget_peft_model(model,\n  lora_config)\nprint_trainable_parameters()\n→ expect <1% trainable"]
  T["4. TRAIN adapters only\nSFTTrainer (TRL)\nor manual loop\n4-bit base never moves"]
  Branch{"5. DEPLOY\nhow?"}

  L --> P --> A --> T --> Branch
  Branch -->|"merge"| Merge["merge_and_unload()\n→ ONE self-contained model\nbest for production deploy\n(can re-quantize to GGUF/AWQ)"]
  Branch -->|"keep separate"| Save["save_pretrained()\n→ adapter_model.safetensors\n(<100 MB)\nbest for hot-swapping\n(one base, many adapters)"]

  style L fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style P fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style T fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Branch fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Merge fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Save fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa

Diagram 5 — Attention-Only vs All-Linear Targeting

Type: Side-by-side comparison Purpose: Why the modern default is all-linear, not attention-only. The single biggest quality lever in the LoRA config. Reading the diagram: Two adapters with similar trainable-param budgets. Attention-only (2021 default) targets q/v_proj — cheap, weaker. All-linear (2024 default) targets all 7 projection modules — stronger, distributes capacity across attention AND the MLP feature pathway.

flowchart TD
  Task["Same steering task\n(format / persona / style)"]

  Task --> Attn["ATTENTION-ONLY\n(2021 LoRA default)\ntarget_modules = ['q_proj','v_proj']"]
  Task --> All["ALL-LINEAR\n(2024 modern default)\ntarget_modules = q,k,v,o_proj\n  + gate,up,down_proj"]

  Attn --> AttnMods["2 modules per layer\nattention pathway only\nMLP features untouched"]
  All --> AllMods["7 modules per layer\nattention + MLP\ndistributes capacity sensibly"]

  AttnMods --> Weak["weaker steer\n(some tasks never land)\nneeds higher r to compensate\n→ approaches all-linear param count\n  for worse quality"]
  AllMods --> Strong["stronger, more reliable\nmatches/beats attn-only\nat comparable param budget\nthe recommended default"]

  Weak --> Verdict{"verdict"}
  Strong --> Verdict
  Verdict -->|"use all-linear"| Default["DEFAULT: all-linear\nunless you have a\nspecific reason not to"]

  style Task fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Attn fill:#14141f,stroke:rgba(240,168,104,0.6),color:#f0a868
  style All fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style AttnMods fill:#08080c,stroke:rgba(240,168,104,0.5),color:#f0a868
  style AllMods fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Weak fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style Strong fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Verdict fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Default fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa

Validation notes

All five diagrams use the course design system colors: #14141f panel fill, #5eead4 (--accent) for the primary/learned path, rgba(94,234,212,0.4) for decision diamonds, #f08080 (--danger) for anti-pattern/loss nodes, #f0a868 (--warn) for the weaker/2021-default path, #82e0aa (--ok) for the recommended/success path, #e4e4e8/#9494a0 for text.
Paste each into Mermaid Live Editor to render. All use stable Mermaid syntax (flowchart LR/TD, decision diamonds {}, dashed -.label.-> annotation links) supported in current Mermaid (v10.4+).
For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.

# Diagrams — Module FT08: LoRA & QLoRA **Module**: FT08 — LoRA & QLoRA **Diagram count**: 5 **Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live). --- ## Diagram 1 — The LoRA Decomposition: W = W₀ + BA **Type**: Mechanism / equation visualization **Purpose**: The one equation the module rests on. W₀ is frozen; B (zero-init) and A (random-init) are tiny trainable matrices that compose into a low-rank update ΔW = BA. **Reading the diagram**: The frozen pretrained W₀ (large, d×k) stays put. A (r×k) and B (d×r) are injected on a side path. Their product BA is added to W₀. B starts at zero so ΔW = 0 at step 0 (identity with the base). ```mermaid flowchart LR X["input x\n(d-vector)"] X --> W0["W₀\nFROZEN\npretrained\n(d × k)\n~16.7M params\n(for 4096²)"] X --> A["A\nTRAINABLE\nrandom-init\n(r × k)\nr=8 → tiny"] A --> B["B\nTRAINABLE\nZERO-init\n(d × r)\nstart = 0"] B --> Scale["× α/r\n(scaling)"] Scale --> BA["ΔW = BA\nlow-rank update"] W0 --> Sum(("+")) BA --> Sum Sum --> Out["output\n= W₀x + (α/r)BAx\nmodel starts == base"] B -.->|"step 0: B=0\n→ ΔW=0\n→ adapter is identity"| Note0["robust: cannot break\nthe base on step 1"] BA -.->|"256× fewer params\nthan full ΔW"| Note1["this is why\n<1% trainable"] style W0 fill:#14141f,stroke:rgba(148,148,160,0.5),stroke-width:1.5px,color:#9494a0 style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style B fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style Scale fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style BA fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style Sum fill:#08080c,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style X fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Note0 fill:#08080c,stroke:rgba(130,224,170,0.4),color:#82e0aa style Note1 fill:#08080c,stroke:rgba(130,224,170,0.4),color:#82e0aa ``` --- ## Diagram 2 — The LoRA Config Parameter Map **Type**: Config-knob reference **Purpose**: The four knobs that define every LoRA adapter, with the modern default for each. **Reading the diagram**: Four knobs (rank, alpha, target modules, dropout). The center node is the resulting `LoraConfig`. Defaults shown are the modern production starting point (r=16, α=32, all-linear, dropout 0.05). ```mermaid flowchart TD Config["LoraConfig\nthe adapter definition"] R["RANK (r)\n# directions of change\nDEFAULT: 16\nlow (8): narrow tasks\nhigh (64): multi-task, watch overfit"] Alpha["ALPHA (α)\nscaling: applied as α/r\nCONVENTION: α ≈ 2×r\n(r=16 → α=32)\ntoo low → base dominates\ntoo high → forgetting"] Tgt["TARGET MODULES\nbiggest quality lever\nDEFAULT: ALL linear\nq,k,v,o_proj + gate,up,down_proj\nNOT attention-only (2021 default)"] Drop["DROPOUT\nregularization on adapter\nDEFAULT: 0.05\nsmall data / high r → raise\nlarge data → 0 ok"] R --> Config Alpha --> Config Tgt --> Config Drop --> Config Config --> PEFT["get_peft_model(model, config)\n→ <1% params trainable\n→ B zero-init: starts == base"] style Config fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#e4e4e8 style R fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Alpha fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Tgt fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Drop fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style PEFT fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa ``` --- ## Diagram 3 — QLoRA's Three Innovations **Type**: Component decomposition **Purpose**: Why QLoRA works — three innovations, each necessary, that compose to fit 7B on a 24GB card. **Reading the diagram**: The frozen base is quantized three ways down. NF4 shrinks the weights to 4-bit optimal-for-Gaussian. Double quant trims the constants. Paged optimizers absorb the checkpointing spikes. Drop any one and the method degrades. ```mermaid flowchart TD Base["7B base weights\nFP16: ~14 GB\n(does NOT fit a 24GB card\nonce you add training overhead)"] Base --> NF4["INNOVATION 1 — NF4\nNormalFloat 4-bit\nquantile bins matched to\nthe NORMAL distribution\n(info-theoretically optimal\nfor Gaussian weights)\n→ 14 GB ≈ 3.5 GB"] NF4 --> DQ["INNOVATION 2 — Double Quantization\nquantize the QUANTIZATION CONSTANTS\nthemselves to 8-bit\nsaves ~0.37 bits/param\n→ trims ~0.3 GB overhead"] DQ --> Paged["INNOVATION 3 — Paged Optimizers\nNVIDIA Unified Memory pages\noptimizer state to CPU on spikes\n(avoids OOM during checkpointing)\n→ absorbs the killer spikes"] NF4 -.without.-> L1["base worse quality\n(uniform 4-bit bins)"] DQ -.without.-> L2["~0.3 GB overhead\n→ may OOM at margin"] Paged -.without.-> L3["steady-state fits\nbut spikes crash the job"] Paged --> Train["LoRA adapters train in BF16\non the 4-bit frozen base\n→ 7B QLoRA: ~10–14 GB\n→ fits RTX 4090 (24GB)"] style Base fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style NF4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style DQ fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style Paged fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style L1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style L2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style L3 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style Train fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa ``` --- ## Diagram 4 — The Full QLoRA Workflow **Type**: Sequential workflow **Purpose**: The five-step loop you run for nearly every steering task. Load 4-bit → prepare → attach → train → merge (or save). **Reading the diagram**: Left to right. Each node is a concrete API call. The branch at the end is the merge-vs-keep-adapter decision. ```mermaid flowchart LR L["1. LOAD base in 4-bit\nBitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_quant_type='nf4',\n bnb_4bit_use_double_quant=True,\n bnb_4bit_compute_dtype=bf16)"] P["2. PREPARE k-bit\nprepare_model_for_\nkbit_training(model)\n(grad checkpointing, etc.)\n—the step everyone forgets"] A["3. ATTACH adapters\nget_peft_model(model,\n lora_config)\nprint_trainable_parameters()\n→ expect <1% trainable"] T["4. TRAIN adapters only\nSFTTrainer (TRL)\nor manual loop\n4-bit base never moves"] Branch{"5. DEPLOY\nhow?"} L --> P --> A --> T --> Branch Branch -->|"merge"| Merge["merge_and_unload()\n→ ONE self-contained model\nbest for production deploy\n(can re-quantize to GGUF/AWQ)"] Branch -->|"keep separate"| Save["save_pretrained()\n→ adapter_model.safetensors\n(<100 MB)\nbest for hot-swapping\n(one base, many adapters)"] style L fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style P fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style T fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Branch fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style Merge fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Save fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa ``` --- ## Diagram 5 — Attention-Only vs All-Linear Targeting **Type**: Side-by-side comparison **Purpose**: Why the modern default is all-linear, not attention-only. The single biggest quality lever in the LoRA config. **Reading the diagram**: Two adapters with similar trainable-param budgets. Attention-only (2021 default) targets q/v_proj — cheap, weaker. All-linear (2024 default) targets all 7 projection modules — stronger, distributes capacity across attention AND the MLP feature pathway. ```mermaid flowchart TD Task["Same steering task\n(format / persona / style)"] Task --> Attn["ATTENTION-ONLY\n(2021 LoRA default)\ntarget_modules = ['q_proj','v_proj']"] Task --> All["ALL-LINEAR\n(2024 modern default)\ntarget_modules = q,k,v,o_proj\n + gate,up,down_proj"] Attn --> AttnMods["2 modules per layer\nattention pathway only\nMLP features untouched"] All --> AllMods["7 modules per layer\nattention + MLP\ndistributes capacity sensibly"] AttnMods --> Weak["weaker steer\n(some tasks never land)\nneeds higher r to compensate\n→ approaches all-linear param count\n for worse quality"] AllMods --> Strong["stronger, more reliable\nmatches/beats attn-only\nat comparable param budget\nthe recommended default"] Weak --> Verdict{"verdict"} Strong --> Verdict Verdict -->|"use all-linear"| Default["DEFAULT: all-linear\nunless you have a\nspecific reason not to"] style Task fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style Attn fill:#14141f,stroke:rgba(240,168,104,0.6),color:#f0a868 style All fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style AttnMods fill:#08080c,stroke:rgba(240,168,104,0.5),color:#f0a868 style AllMods fill:#14141f,stroke:#5eead4,color:#e4e4e8 style Weak fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style Strong fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Verdict fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style Default fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa ``` --- ## Validation notes - All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` (`--accent`) for the primary/learned path, `rgba(94,234,212,0.4)` for decision diamonds, `#f08080` (`--danger`) for anti-pattern/loss nodes, `#f0a868` (`--warn`) for the weaker/2021-default path, `#82e0aa` (`--ok`) for the recommended/success path, `#e4e4e8`/`#9494a0` for text. - Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart LR/TD`, decision diamonds `{}`, dashed `-.label.->` annotation links) supported in current Mermaid (v10.4+). - For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.