Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT08 — LoRA & QLoRA Duration: 20–30 minutes of GPU time (plus ~10 min setup) on a consumer GPU or Colab T4 Environment: Python 3.10+, a CUDA GPU with ≥16 GB VRAM (RTX 4090 / 3090 / A10G / Colab T4 all work). Apple Silicon (M-series, ≥16 GB) works for the smaller base via MPS — see the fallback note in Phase 0.
This is your first real fine-tune. By the end you will have loaded a 1–1.5B base in 4-bit, attached a LoRA adapter, trained it on 500 style-steering examples, merged the adapter into the base, and run inference to confirm the steer took. The full loop — the one you will repeat for nearly every steering task in the rest of the course.
python3 -m venv ft08-env && source ft08-env/bin/activate
pip install -q -U "torch>=2.3" "transformers>=4.46" "peft>=0.13" \
"trl>=0.12" "bitsandbytes>=0.43" "accelerate>=0.34" "datasets>=3.0"
# On Apple Silicon (no CUDA), drop bitsandbytes — you will use the MPS fallback (Phase 6).
On Colab: Runtime → Change runtime type → T4 GPU. Then run the same pip install (drop the venv lines).
You do not need a Hugging Face token for the default base (Qwen/Qwen2.5-1.5B-Instruct is open). If you prefer meta-llama/Llama-3.2-1B-Instruct or openbmb/MiniCPM3-1B, accept the model license and set HF_TOKEN — they are gated.
By the end of this lab you will have:
The default for this lab is Qwen/Qwen2.5-1.5B-Instruct — open (no gating), a clean ChatML template, the richest QLoRA ecosystem, and it fits comfortably on a free Colab T4. The two documented one-line swaps:
| Base | Why | Notes |
|---|---|---|
Qwen/Qwen2.5-1.5B-Instruct |
Default. Open, no gating, ChatML, ~3 GB at 4-bit. | Best for Colab T4. |
openbmb/MiniCPM5-1B |
Smaller, open-data lineage (FT02). | Slightly different template; same config works. |
meta-llama/Llama-3.2-1B-Instruct |
Llama family; header-id template (FT07). | Gated — needs HF_TOKEN. |
Set the model id once at the top of your script so you can re-run with any of them:
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct" # default; swap freely
We steer the model to respond in a specific style: always answer as a terse, contrarian pirate ("Ar, no — that's the wrong question..."). This is a pure steering task (FT00): the base already knows pirate vocabulary; we are making it reliable and in our format. It is the ideal first fine-tune — unambiguous success criterion, no knowledge injection.
Create make_data.py:
# make_data.py — generate 500 style-steering examples
import json, random
random.seed(42)
QUESTIONS = [
"What is machine learning?",
"Explain recursion.",
"How does a transformer work?",
"What is gradient descent?",
"Why is the sky blue?",
"What is a neural network?",
"Explain tokenization.",
"What is backpropagation?",
"How does photosynthesis work?",
"What is the capital of France?",
"Explain object-oriented programming.",
"What is a database index?",
"How do vaccines work?",
"What is the Internet?",
"Explain the theory of relativity.",
]
def pirate_reply(q: str) -> str:
# A terse, contrarian pirate persona. The base already knows this vocabulary;
# we are steering it to be reliable and in-format.
openers = ["Ar, no —", "Bah,", "Nay,", "Gar,", "Yarr, but —"]
middles = [
f"ye're askin' the wrong question. The real o' it be: {q.lower().rstrip('?')}",
f"that's a landlubber's tale. The truth be murkier than ye think.",
f"I've seen stranger on the high seas. The short answer: aye, mostly.",
f"don't trust the book-learned on this. Trust the salt in yer bones.",
f"every lubber asks that. The answer changes with the tide.",
]
closers = [" Now fetch the rum.", " Savvy?", " Arr.", " Off with ye.", ""]
return f"{random.choice(openers)} {random.choice(middles)}{random.choice(closers)}"
rows = []
for _ in range(500):
q = random.choice(QUESTIONS)
rows.append({
"messages": [
{"role": "system", "content": "You are a terse, contrarian pirate. You always answer in pirate speak, briefly, and you push back on the question."},
{"role": "user", "content": q},
{"role": "assistant", "content": pirate_reply(q)},
]
})
with open("pirate_sft.jsonl", "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} examples to pirate_sft.jsonl")
Run it:
python make_data.py
head -1 pirate_sft.jsonl | python -m json.tool
Read one example. This is your steering target. The lab will fail honestly if the model does not adopt this style.
FT07 discipline checkpoint. Before training, run one example through
apply_chat_template(..., return_assistant_tokens_mask=True)and decode it. Confirm the ChatML role tokens and EOS are present. You already learned why (FT07). Do not skip it this time either.
Create train.py. We load the base quantized to NF4 with double quantization on — QLoRA innovation 1 and 2.
# train.py — QLoRA fine-tune (Phases 2–4)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
# --- QLoRA innovations 1 (NF4) + 2 (double quant) ---
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat 4-bit
bnb_4bit_use_double_quant=True, # quantize the constants too
bnb_4bit_compute_dtype=torch.bfloat16, # compute in bf16
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token_id is None:
tokenizer.pad_token = tokenizer.eos_token # Qwen ChatML: pad == eos often
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
quantization_config=bnb_config,
device_map="auto",
attn_implementation="flash_attention_2", # FT01: effectively mandatory
)
If FlashAttention 2 is unavailable (e.g., older GPU, Colab T4 fallback), use attn_implementation="sdpa" — slower but correct.
The step everyone forgets (prepare_model_for_kbit_training) and the modern all-linear LoRA config.
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
# 1. PREPARE the 4-bit base for adapter training (the forgotten step)
model = prepare_model_for_kbit_training(model)
# 2. ATTACH adapters — modern default: ALL attention + ALL MLP projections
lora_config = LoraConfig(
r=16,
lora_alpha=32, # convention: alpha ≈ 2×r
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj", # attention
"gate_proj", "up_proj", "down_proj", # MLP (modern default)
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
# Sanity check — you MUST see <1% trainable params
model.print_trainable_parameters()
You should see output like trainable params: 9,437,696 || all params: 1,577,778,688 || trainable%: 0.5982. Under 1% — exactly the LoRA promise. If you see a large trainable%, you forgot to attach the adapter or the config is wrong.
We use TRL's SFTTrainer with the FT01 knobs (gradient checkpointing, paged optimizer). The chat template is applied automatically because we use the messages format — TRL/transformers calls apply_chat_template under the hood.
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
dataset = load_dataset("json", data_files="pirate_sft.jsonl", split="train")
# FT01 knobs: grad checkpointing, paged 8-bit optimizer, FA2 already on the model
training_args = SFTConfig(
output_dir="./qlora-pirate",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=4, # effective batch = 8
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
logging_steps=5,
save_strategy="epoch",
bf16=True, # use fp16=True on older cards (V100)
gradient_checkpointing=True,
optim="paged_adamw_8bit", # QLoRA innovation 3 (paged optimizers)
max_seq_length=512, # our examples are short
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
processing_class=tokenizer,
)
trainer.train()
Watch the loss. It should descend smoothly from ~1.5–2.0 to ~0.8–1.2 over ~190 steps (500 × 3 epochs / 8 effective batch). If it is NaN or exploding, stop — that is an FT07 template/EOS bug, not a QLoRA bug. Re-run the FT07 inspection loop on one tokenized example.
Watch the VRAM with nvidia-smi -l 2 in a second terminal. Peak should land around 6–10 GB for a 1.5B QLoRA at 512 context — exactly where the FT01 rules of thumb said. If you OOM, lower per_device_train_batch_size to 1 and raise gradient_accumulation_steps to 8 (same effective batch).
Two options. We do both so you can see each artifact.
# --- Option A: SAVE the adapter only (hot-swappable, <100 MB) ---
model.save_pretrained("./qlora-pirate/adapter-only")
tokenizer.save_pretrained("./qlora-pirate/adapter-only")
# Later: PeftModel.from_pretrained(base, "./qlora-pirate/adapter-only")
# --- Option B: MERGE the adapter into the base (self-contained, for deploy) ---
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./qlora-pirate/merged", safe_serialization=True)
tokenizer.save_pretrained("./qlora-pirate/merged")
Option B produces a single model directory you can load directly with AutoModelForCausalLM.from_pretrained("./qlora-pirate/merged") — no PEFT, no 4-bit, no adapter at serve time. This is what you would quantize to GGUF for Ollama (FT19) or serve in vLLM (FT20).
Caveat:
merge_and_unload()on a 4-bit base produces a model whose weights are the dequantized merge — it will be larger than the original 4-bit base. For production you re-quantize the merged model (GGUF/AWQ) afterward. This is correct and expected; the merge bakes in the steer, the re-quantize compresses for deployment.
The moment of truth. Load the merged model and ask it a question the training set never contained:
# infer.py — verify the steer
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_PATH = "./qlora-pirate/merged"
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(
MODEL_PATH, torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [
{"role": "system", "content": "You are a terse, contrarian pirate. You always answer in pirate speak, briefly, and you push back on the question."},
{"role": "user", "content": "What's the best programming language for beginners?"}, # NOT in training set
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(inputs, max_new_tokens=80, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
Expected: a terse, contrarian pirate reply — "Ar, no —", "Bah,", pushing back on the question — even though that exact question was never in the training set. The steer generalized. This is the FT00 thesis in action: the base already knew pirate vocabulary; you steered it to use it reliably and in your format.
Control: load the original base (Qwen/Qwen2.5-1.5B-Instruct) with the same system prompt and the same question. It will be polite and helpful, not a contrarian pirate. The difference between the two outputs is your adapter. That gap is what 0.6% of the parameters bought you.
Submit ft08-lab-report.md containing:
make_data.py and a sample of 3 generated examples (the steering target).print_trainable_parameters() output — confirm trainable% < 1. Report the exact number.log_history JSON). Report start loss, end loss, and step count.nvidia-smi). Compare to the FT01 rule of thumb for a 1.5B QLoRA (~1.5–2× the 4-bit base ≈ 1 GB weights + ~5–8 GB overhead).These are defensible answers, not the only wording.
For Qwen2.5-1.5B at r=16, all-linear: ~9.4M trainable / ~1.58B total ≈ 0.60%. (Exact number varies a hair by transformers/peft version.) Under 1% — the LoRA promise holds.
Expect: start ~1.6–2.0, smooth cosine descent, end ~0.8–1.1 after 3 epochs (~190 steps at effective batch 8). If loss plateaus above ~1.4, the rank may be too low or the LR too low; if it crashes below ~0.6, you are overfitting (raise dropout or reduce epochs). A healthy run lands in the ~0.8–1.1 band and generalizes (Phase 6 confirms).
Peak ~6–10 GB on a 1.5B QLoRA at 512 context, batch 2 + grad accum 4. The FT01 rule of thumb ("~1.5–2× the 4-bit model size plus overhead") gives 4-bit 1.5B ≈ 0.75 GB → ~1.5 GB shorthand, plus the real activation/optimizer overhead of ~5–8 GB → ~7–10 GB. Matches observation. On a Colab T4 (16 GB) you have comfortable headroom.
Merged model outputs should be recognizably pirate, terse, and contrarian on unseen questions. The base with the same system prompt will be polite and helpful. The gap between these two is your adapter. If both look identical, the steer did not take — re-check that you merged the trained adapter (not an untrained one) and that loss actually descended.
Under 1% of the parameters sufficed because steering is a low-rank operation. The base already knew pirate vocabulary and the contrarian stance from pretraining — we were not teaching it anything new, we were redirecting its probability mass so it uses that vocabulary reliably and in our format. The intrinsic dimension hypothesis (Aghajanyan) predicts exactly this: the useful changes during fine-tuning live in a low-rank subspace, so a tiny adapter (the B·A pair at each targeted layer) can express them. This is the FT00 thesis made operational — fine-tuning steers behavior; it does not teach knowledge — and the 0.6% trainable count is the numerical proof that the task was steering, not knowledge injection. Had it required full-FT-scale updates, that would have been evidence we were trying to teach, not steer.
target_modules=["q_proj","v_proj"] only, at the same rank. Compare to the all-linear run. Quantify the quality gap on the same 5 questions. This is the FT08 "biggest quality lever" claim, felt directly.llama.cpp's convert_hf_to_gguf.py, then load it in Ollama. You have gone from "base in 4-bit" to "merged, re-quantized, served locally" — the full Layer 2 → Layer 4 path of the FT00 steering stack.# Lab Specification — Module FT08: LoRA & QLoRA
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT08 — LoRA & QLoRA
**Duration**: 20–30 minutes of GPU time (plus ~10 min setup) on a consumer GPU or Colab T4
**Environment**: Python 3.10+, a CUDA GPU with ≥16 GB VRAM (RTX 4090 / 3090 / A10G / Colab T4 all work). Apple Silicon (M-series, ≥16 GB) works for the smaller base via MPS — see the fallback note in Phase 0.
> **This is your first real fine-tune.** By the end you will have loaded a 1–1.5B base in 4-bit, attached a LoRA adapter, trained it on 500 style-steering examples, merged the adapter into the base, and run inference to confirm the steer took. The full loop — the one you will repeat for nearly every steering task in the rest of the course.
---
## Setup (one time)
```bash
python3 -m venv ft08-env && source ft08-env/bin/activate
pip install -q -U "torch>=2.3" "transformers>=4.46" "peft>=0.13" \
"trl>=0.12" "bitsandbytes>=0.43" "accelerate>=0.34" "datasets>=3.0"
# On Apple Silicon (no CUDA), drop bitsandbytes — you will use the MPS fallback (Phase 6).
```
On Colab: Runtime → Change runtime type → T4 GPU. Then run the same `pip install` (drop the venv lines).
You do **not** need a Hugging Face token for the default base (`Qwen/Qwen2.5-1.5B-Instruct` is open). If you prefer `meta-llama/Llama-3.2-1B-Instruct` or `openbmb/MiniCPM3-1B`, accept the model license and set `HF_TOKEN` — they are gated.
---
## Learning objectives
By the end of this lab you will have:
1. **Executed the full QLoRA workflow** — load 4-bit → prepare k-bit → attach LoRA → train adapters → merge — end to end, on a real base model, producing a self-contained merged checkpoint.
2. **Configured a production LoRA adapter** from the four knobs (rank, alpha, target modules, dropout), using the modern all-linear default, and verified the trainable-param count is under 1%.
3. **Verified the steer took** by running inference on the merged model and observing the style shift (the base did not respond in that style; the steered model does).
4. **Felt the FT01 VRAM math** — you will see ~6–10 GB peak on a 1.5B model, exactly where the rules of thumb said it would land.
5. **Internalized the loop** so that every subsequent module (FT09 DoRA, FT11 the training loop, FT12 SFT) builds on a workflow you have already run, not one you are reading about.
---
## Phase 0 — Pick your base (2 min)
The default for this lab is **`Qwen/Qwen2.5-1.5B-Instruct`** — open (no gating), a clean ChatML template, the richest QLoRA ecosystem, and it fits comfortably on a free Colab T4. The two documented one-line swaps:
| Base | Why | Notes |
| --- | --- | --- |
| `Qwen/Qwen2.5-1.5B-Instruct` | **Default.** Open, no gating, ChatML, ~3 GB at 4-bit. | Best for Colab T4. |
| `openbmb/MiniCPM5-1B` | Smaller, open-data lineage (FT02). | Slightly different template; same config works. |
| `meta-llama/Llama-3.2-1B-Instruct` | Llama family; header-id template (FT07). | Gated — needs `HF_TOKEN`. |
Set the model id once at the top of your script so you can re-run with any of them:
```python
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct" # default; swap freely
```
---
## Phase 1 — Build the style-steering dataset (5 min)
We steer the model to respond in a specific style: **always answer as a terse, contrarian pirate** ("Ar, no — that's the wrong question..."). This is a pure steering task (FT00): the base already knows pirate vocabulary; we are making it *reliable* and *in our format*. It is the ideal first fine-tune — unambiguous success criterion, no knowledge injection.
Create `make_data.py`:
```python
# make_data.py — generate 500 style-steering examples
import json, random
random.seed(42)
QUESTIONS = [
"What is machine learning?",
"Explain recursion.",
"How does a transformer work?",
"What is gradient descent?",
"Why is the sky blue?",
"What is a neural network?",
"Explain tokenization.",
"What is backpropagation?",
"How does photosynthesis work?",
"What is the capital of France?",
"Explain object-oriented programming.",
"What is a database index?",
"How do vaccines work?",
"What is the Internet?",
"Explain the theory of relativity.",
]
def pirate_reply(q: str) -> str:
# A terse, contrarian pirate persona. The base already knows this vocabulary;
# we are steering it to be reliable and in-format.
openers = ["Ar, no —", "Bah,", "Nay,", "Gar,", "Yarr, but —"]
middles = [
f"ye're askin' the wrong question. The real o' it be: {q.lower().rstrip('?')}",
f"that's a landlubber's tale. The truth be murkier than ye think.",
f"I've seen stranger on the high seas. The short answer: aye, mostly.",
f"don't trust the book-learned on this. Trust the salt in yer bones.",
f"every lubber asks that. The answer changes with the tide.",
]
closers = [" Now fetch the rum.", " Savvy?", " Arr.", " Off with ye.", ""]
return f"{random.choice(openers)} {random.choice(middles)}{random.choice(closers)}"
rows = []
for _ in range(500):
q = random.choice(QUESTIONS)
rows.append({
"messages": [
{"role": "system", "content": "You are a terse, contrarian pirate. You always answer in pirate speak, briefly, and you push back on the question."},
{"role": "user", "content": q},
{"role": "assistant", "content": pirate_reply(q)},
]
})
with open("pirate_sft.jsonl", "w") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} examples to pirate_sft.jsonl")
```
Run it:
```bash
python make_data.py
head -1 pirate_sft.jsonl | python -m json.tool
```
Read one example. This is your steering target. The lab will fail honestly if the model does not adopt this style.
> **FT07 discipline checkpoint.** Before training, run one example through `apply_chat_template(..., return_assistant_tokens_mask=True)` and decode it. Confirm the ChatML role tokens and EOS are present. You already learned why (FT07). Do not skip it this time either.
---
## Phase 2 — Load the base in 4-bit (3 min)
Create `train.py`. We load the base quantized to NF4 with double quantization on — QLoRA innovation 1 and 2.
```python
# train.py — QLoRA fine-tune (Phases 2–4)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
# --- QLoRA innovations 1 (NF4) + 2 (double quant) ---
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat 4-bit
bnb_4bit_use_double_quant=True, # quantize the constants too
bnb_4bit_compute_dtype=torch.bfloat16, # compute in bf16
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token_id is None:
tokenizer.pad_token = tokenizer.eos_token # Qwen ChatML: pad == eos often
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
quantization_config=bnb_config,
device_map="auto",
attn_implementation="flash_attention_2", # FT01: effectively mandatory
)
```
If FlashAttention 2 is unavailable (e.g., older GPU, Colab T4 fallback), use `attn_implementation="sdpa"` — slower but correct.
---
## Phase 3 — Prepare k-bit + attach LoRA (3 min)
The step everyone forgets (`prepare_model_for_kbit_training`) and the modern all-linear LoRA config.
```python
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
# 1. PREPARE the 4-bit base for adapter training (the forgotten step)
model = prepare_model_for_kbit_training(model)
# 2. ATTACH adapters — modern default: ALL attention + ALL MLP projections
lora_config = LoraConfig(
r=16,
lora_alpha=32, # convention: alpha ≈ 2×r
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj", # attention
"gate_proj", "up_proj", "down_proj", # MLP (modern default)
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
# Sanity check — you MUST see <1% trainable params
model.print_trainable_parameters()
```
You should see output like `trainable params: 9,437,696 || all params: 1,577,778,688 || trainable%: 0.5982`. **Under 1% — exactly the LoRA promise.** If you see a large trainable%, you forgot to attach the adapter or the config is wrong.
---
## Phase 4 — Train the adapters (8–15 min GPU)
We use TRL's `SFTTrainer` with the FT01 knobs (gradient checkpointing, paged optimizer). The chat template is applied automatically because we use the `messages` format — TRL/transformers calls `apply_chat_template` under the hood.
```python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
dataset = load_dataset("json", data_files="pirate_sft.jsonl", split="train")
# FT01 knobs: grad checkpointing, paged 8-bit optimizer, FA2 already on the model
training_args = SFTConfig(
output_dir="./qlora-pirate",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=4, # effective batch = 8
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
logging_steps=5,
save_strategy="epoch",
bf16=True, # use fp16=True on older cards (V100)
gradient_checkpointing=True,
optim="paged_adamw_8bit", # QLoRA innovation 3 (paged optimizers)
max_seq_length=512, # our examples are short
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
processing_class=tokenizer,
)
trainer.train()
```
**Watch the loss.** It should descend smoothly from ~1.5–2.0 to ~0.8–1.2 over ~190 steps (500 × 3 epochs / 8 effective batch). If it is NaN or exploding, stop — that is an FT07 template/EOS bug, not a QLoRA bug. Re-run the FT07 inspection loop on one tokenized example.
**Watch the VRAM** with `nvidia-smi -l 2` in a second terminal. Peak should land around **6–10 GB** for a 1.5B QLoRA at 512 context — exactly where the FT01 rules of thumb said. If you OOM, lower `per_device_train_batch_size` to 1 and raise `gradient_accumulation_steps` to 8 (same effective batch).
---
## Phase 5 — Merge and save (2 min)
Two options. We do **both** so you can see each artifact.
```python
# --- Option A: SAVE the adapter only (hot-swappable, <100 MB) ---
model.save_pretrained("./qlora-pirate/adapter-only")
tokenizer.save_pretrained("./qlora-pirate/adapter-only")
# Later: PeftModel.from_pretrained(base, "./qlora-pirate/adapter-only")
# --- Option B: MERGE the adapter into the base (self-contained, for deploy) ---
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./qlora-pirate/merged", safe_serialization=True)
tokenizer.save_pretrained("./qlora-pirate/merged")
```
Option B produces a single model directory you can load directly with `AutoModelForCausalLM.from_pretrained("./qlora-pirate/merged")` — no PEFT, no 4-bit, no adapter at serve time. This is what you would quantize to GGUF for Ollama (FT19) or serve in vLLM (FT20).
> **Caveat:** `merge_and_unload()` on a 4-bit base produces a model whose weights are the *dequantized* merge — it will be larger than the original 4-bit base. For production you re-quantize the merged model (GGUF/AWQ) afterward. This is correct and expected; the merge bakes in the steer, the re-quantize compresses for deployment.
---
## Phase 6 — Run inference and confirm the steer took (3 min)
The moment of truth. Load the merged model and ask it a question the training set never contained:
```python
# infer.py — verify the steer
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_PATH = "./qlora-pirate/merged"
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(
MODEL_PATH, torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [
{"role": "system", "content": "You are a terse, contrarian pirate. You always answer in pirate speak, briefly, and you push back on the question."},
{"role": "user", "content": "What's the best programming language for beginners?"}, # NOT in training set
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(inputs, max_new_tokens=80, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```
**Expected:** a terse, contrarian pirate reply — "Ar, no —", "Bah,", pushing back on the question — even though that exact question was never in the training set. **The steer generalized.** This is the FT00 thesis in action: the base already knew pirate vocabulary; you steered it to use it reliably and in your format.
**Control:** load the original base (`Qwen/Qwen2.5-1.5B-Instruct`) with the same system prompt and the same question. It will be polite and helpful, not a contrarian pirate. The difference between the two outputs *is* your adapter. That gap is what 0.6% of the parameters bought you.
---
## Deliverables
Submit `ft08-lab-report.md` containing:
- [ ] **Your `make_data.py`** and a sample of 3 generated examples (the steering target).
- [ ] **The FT07 inspection output** for one tokenized example (decoded, EOS present, assistant mask correct).
- [ ] **`print_trainable_parameters()` output** — confirm trainable% < 1. Report the exact number.
- [ ] **The training loss curve** (TensorBoard screenshot or the `log_history` JSON). Report start loss, end loss, and step count.
- [ ] **Peak VRAM observed** (`nvidia-smi`). Compare to the FT01 rule of thumb for a 1.5B QLoRA (~1.5–2× the 4-bit base ≈ 1 GB weights + ~5–8 GB overhead).
- [ ] **Three inference outputs** from the merged model on questions NOT in the training set, plus the same three from the un-steered base. The style difference is your evidence the steer took.
- [ ] A 4–6 sentence reflection: why did <1% of the parameters suffice to produce a reliable style shift? Tie it to the FT00 thesis and the intrinsic dimension hypothesis.
---
## Solution key
These are defensible answers, not the only wording.
### Trainable params
For Qwen2.5-1.5B at r=16, all-linear: **~9.4M trainable / ~1.58B total ≈ 0.60%**. (Exact number varies a hair by transformers/peft version.) Under 1% — the LoRA promise holds.
### Loss trajectory
Expect: start ~1.6–2.0, smooth cosine descent, end ~0.8–1.1 after 3 epochs (~190 steps at effective batch 8). If loss plateaus above ~1.4, the rank may be too low or the LR too low; if it crashes below ~0.6, you are overfitting (raise dropout or reduce epochs). A healthy run lands in the ~0.8–1.1 band and *generalizes* (Phase 6 confirms).
### VRAM
Peak ~6–10 GB on a 1.5B QLoRA at 512 context, batch 2 + grad accum 4. The FT01 rule of thumb ("~1.5–2× the 4-bit model size plus overhead") gives 4-bit 1.5B ≈ 0.75 GB → ~1.5 GB shorthand, plus the real activation/optimizer overhead of ~5–8 GB → ~7–10 GB. Matches observation. On a Colab T4 (16 GB) you have comfortable headroom.
### Inference evidence
Merged model outputs should be recognizably pirate, terse, and contrarian on unseen questions. The base with the same system prompt will be polite and helpful. **The gap between these two is your adapter.** If both look identical, the steer did not take — re-check that you merged the *trained* adapter (not an untrained one) and that loss actually descended.
### Reflection (model answer)
Under 1% of the parameters sufficed because steering is a low-rank operation. The base already knew pirate vocabulary and the contrarian stance from pretraining — we were not teaching it anything new, we were redirecting its probability mass so it uses that vocabulary reliably and in our format. The intrinsic dimension hypothesis (Aghajanyan) predicts exactly this: the useful changes during fine-tuning live in a low-rank subspace, so a tiny adapter (the B·A pair at each targeted layer) can express them. This is the FT00 thesis made operational — *fine-tuning steers behavior; it does not teach knowledge* — and the 0.6% trainable count is the numerical proof that the task was steering, not knowledge injection. Had it required full-FT-scale updates, that would have been evidence we were trying to teach, not steer.
---
## Stretch goals
1. **Rank sweep.** Re-train at r=4, r=16, r=64 (keep α=2×r). Compare the merged models' outputs on a fixed set of 5 unseen questions. Where does r=4 underfit (the style never quite lands)? Where does r=64 overfit (repetitive, degraded coherence)? Find your knee.
2. **Attention-only vs all-linear.** Re-train with `target_modules=["q_proj","v_proj"]` only, at the same rank. Compare to the all-linear run. Quantify the quality gap on the same 5 questions. This is the FT08 "biggest quality lever" claim, felt directly.
3. **Alpha mismatch.** Train with α=8 (half the convention) at r=16. Observe the adapter "speaking too quietly" — the base dominates and the steer is weak. Then α=64 (4×) and observe overcorrection / forgetting. The convention α≈2×r exists for a reason.
4. **Apple Silicon fallback.** On an M-series Mac (≥16 GB), repeat the lab without 4-bit quantization (load the base at bf16 on MPS, attach LoRA, train). A 1.5B base fits on a 16 GB Mac at bf16. You lose the QLoRA innovations but keep the LoRA workflow. Useful for iterating without a CUDA card.
5. **Export to GGUF.** Take the merged model from Option B and convert it to GGUF (FT19 preview) using `llama.cpp`'s `convert_hf_to_gguf.py`, then load it in Ollama. You have gone from "base in 4-bit" to "merged, re-quantized, served locally" — the full Layer 2 → Layer 4 path of the FT00 steering stack.