Fine-Tuning LLMs: LoRA, QLoRA & PEFT

Fine-tuning is powerful, but it is the last tool you should reach for, not the first. Most problems are solved more cheaply by writing a better prompt or by retrieval. Use this rule of thumb:

• Prompting: change the model's output by changing the instruction. Zero training, instant to iterate. Try this first, always.
• RAG (retrieval-augmented generation): the model needs facts it doesn't have — your docs, today's prices, a private wiki. You fetch the relevant text and paste it into the prompt at query time. No training; facts can change every minute.
• Fine-tuning: the model needs a new behaviour — a consistent tone, a strict output format, a niche skill — that prompting can't make reliable. You change the weights with many examples.

The runnable example below builds a tiny decision helper and shows how few parameters LoRA actually trains. Run it and read every comment.

Full fine-tuning updates every weight in the model. It gives the best quality but is brutally expensive: training a 70B model in 32-bit needs hundreds of gigabytes of GPU memory just to hold the weights, gradients, and optimiser state.

PEFT (Parameter-Efficient Fine-Tuning) is the family of methods that avoid this. The most popular is LoRA (Low-Rank Adaptation): freeze the original weights and learn two small matrices whose product is added to a weight matrix. A full weight matrix holds d×d numbers; LoRA trains only 2×d×r, where the rank r (usually 8–16) is tiny. That is often under 1% of the parameters.

QLoRA stacks one more trick on top: load the frozen base model in 4-bit precision (NormalFloat4) instead of 16-bit. The adapters stay in higher precision and do the learning, while the giant frozen base barely takes up room — so a 70B model that needed four A100s can be fine-tuned on a single 48GB GPU.

Here is what a real LoRA setup looks like with Hugging Face peft and transformers. It's read-only (it needs a GPU and the libraries), but notice the trainable% line in the expected output:

# Real LoRA fine-tuning with Hugging Face (read-only — needs a GPU + libraries).
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, TrainingArguments, Trainer

model = AutoModelForCausalLM.from_pretrained("base-llm-7b")

lora = LoraConfig(
    r=8,                 # rank — 8-16 is plenty for most tasks
    lora_alpha=16,       # scaling factor (often 2 * r)
    target_modules=["q_proj", "v_proj"],  # apply to attention Q and V
    lora_dropout=0.05,
)

model = get_peft_model(model, lora)
model.print_trainable_parameters()
# Expected output:
# trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622

trainer = Trainer(
    model=model,
    args=TrainingArguments(output_dir="out", num_train_epochs=3,
                           per_device_train_batch_size=4, learning_rate=2e-4),
    train_dataset=dataset,      # your instruction/response records
)
trainer.train()
model.save_pretrained("my-lora-adapter")   # only ~16MB of adapter weights!

And QLoRA is the same idea with one extra config object to quantise the base model to 4-bit:

# QLoRA: load the base model in 4-bit so a 70B model fits on one GPU.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,                 # 4 bits per weight instead of 16/32
    bnb_4bit_quant_type="nf4",         # NormalFloat4 — best for model weights
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained("base-llm-70b", quantization_config=bnb)
model = get_peft_model(model, LoraConfig(r=16, target_modules=["q_proj", "v_proj"]))

# Memory for the FROZEN base weights at different precisions (70B params):
#   fp32 -> 280 GB   fp16 -> 140 GB   int8 -> 70 GB   int4 -> 35 GB
# Expected output (model.print_trainable_parameters()):
# trainable params: 8,388,608 || all params: 70,000,000,000 || trainable%: 0.0120
#
# Result: a 70B model that needed 4x A100s now fine-tunes on ONE 48GB GPU.

Instruction tuning is just supervised fine-tuning on instruction → response pairs. It teaches the model to follow commands rather than only continue text. This is the workhorse you'll use most.

RLHF (Reinforcement Learning from Human Feedback) goes further to make answers preferred by people. Humans rank several model answers; a small reward model learns to predict those rankings; then the LLM is optimised to score highly under that reward model. It works well but has many moving parts.

DPO (Direct Preference Optimisation) reaches a similar place far more simply. You give it pairs of a chosen answer and a rejected answer, and it trains the model directly to prefer the chosen one — no separate reward model, no reinforcement-learning loop. For most teams today, DPO is the easier path to alignment.

A fine-tune is only as good as its data. You build a list of records, each with a prompt (the instruction, wrapped in a consistent template) and a completion (the ideal response). The template matters: the model learns to produce whatever follows your ### Response: marker, so it must be identical in training and at inference.

Good data is consistent (same format every row), clean (no empty or contradictory answers), and representative (covers the cases you'll actually see). The runnable example below formats raw pairs into records and then shows the core idea of training — a single weight learning by gradient descent. Run it:

# Fine-tuning starts with DATA: instruction -> response pairs.
# Here we format raw examples into clean training records — no libraries needed.

raw_examples = [
    ("Translate to French: Hello", "Bonjour"),
    ("Summarise: The cat sat on the mat.", "A cat sat on a mat."),
    ("Capital of Japan?", "Tokyo"),
]

# A simple chat-style template the model learns to complete.
def to_record(instruction, response):
    return {
        "prompt": "### Instruction:\n" + instruction + "\n\n### Response:
...

That five-line loop is the whole secret of training, scaled up: predict, measure the error, nudge each weight a little against the error, repeat. Fine-tuning runs this over billions of weights (or, with LoRA, just the adapter's few million).

Always hold back a validation set the model never trains on. During training you watch two numbers: training loss (error on data it sees) and validation loss (error on data it doesn't). When training loss keeps falling but validation loss starts rising, you are overfitting — the model is memorising your examples instead of learning the pattern.

• Train for few epochs (often 1–3). More passes mostly memorise.
• Use a small learning rate (e.g. 2e-4 for LoRA) so you nudge, not shove.
• Watch for catastrophic forgetting: if the model gets great at your task but worse at everything else, you trained too hard or on too narrow a dataset.
• Judge on real outputs, not just loss — run the model on held-out prompts and read the answers.