Model Compression

A trained model is just a big pile of numbers (its weights). The more numbers and the more bits each one uses, the more memory it eats and the slower it runs. Compression trades a sliver of accuracy for three things you nearly always need in production:

• Edge devices — a phone, watch, or browser has a few GB of RAM and no datacentre GPU. A 280GB model simply won't load; a 4GB one will.
• Cost — smaller models fit on cheaper hardware. A LLaMA-70B model in FP32 needs four A100 GPUs; in INT4 it fits on a single 24GB card.
• Speed — fewer bytes to move and integer maths to run means lower latency (time per request), which users feel directly.

Most models store each weight as a 32-bit float (FP32) — 4 bytes that can represent tiny fractions very precisely. Quantization asks: do you really need that precision? Often you can map those floats onto 8-bit integers (INT8) — just 256 possible values, 1 byte each. That is an instant 4x size cut, and integer maths runs faster on most CPUs and accelerators.

The trick is a single scale factor. You find the largest weight, divide the range into 256 steps, and store which step each weight is closest to. To use the model you multiply the integer back by the scale to get an approximate float. The gap between the original and the recovered float is the quantization error — small if your weights are well-behaved.

Going further — INT4 (16 levels) gives 8x compression but a bigger error, so it needs smart schemes like GPTQ or AWQ. The two ways to apply quantization: post-training (PTQ) quantizes an already-trained model (easy, slight loss), while quantization-aware training (QAT) simulates the rounding during training so the model learns to tolerate it (better, more work).

Worked example — one-line INT8 quantization in PyTorch:

# Post-training dynamic quantization in PyTorch.
# Maps FP32 weights to INT8 with one line — 4x smaller, faster on CPU.

import torch
import torch.nn as nn

# A small model standing in for any trained network.
model = nn.Sequential(
    nn.Linear(512, 512),
    nn.ReLU(),
    nn.Linear(512, 10),
)
model.eval()

# Quantize the Linear layers to INT8. Dynamic = activations are
# quantized on the fly at inference, weights are stored as INT8.
quantized = torch.quantization.quantize_dynamic(
    model, {nn.Linear}, dtype=torch.qint8
)

def size_mb(m):
    torch.save(m.state_dict(), "tmp.pt")
    import os
    return os.path.getsize("tmp.pt") / 1e6

print(f"FP32 size: {size_mb(model):.2f} MB")
print(f"INT8 size: {size_mb(quantized):.2f} MB")

# Expected output (approximate):
#   FP32 size: 1.07 MB
#   INT8 size: 0.28 MB

The INT8 state dict is roughly a quarter of the FP32 one — exactly the 4x you'd expect from 4 bytes down to 1. You'll build this scale-and-round logic by hand in the runnable example below.

Trained networks are wasteful: a large fraction of weights are so close to zero they contribute almost nothing. Pruning deletes them. The simplest recipe is magnitude pruning — pick a threshold, then zero out every weight whose absolute value is below it. The fraction of weights now zero is the sparsity.

There is a crucial catch. Magnitude (unstructured) pruning scatters zeros anywhere in the matrix — you get high sparsity, but the matrix is still the same shape, so ordinary GPUs do the same amount of work and you get no speedup, only a smaller file (when stored sparsely). Structured pruning instead removes whole rows, channels, or attention heads, so the matrix genuinely shrinks and standard hardware runs it faster. Rule of thumb: unstructured for size, structured for latency.

Rather than shrink a model, knowledge distillation trains a brand-new small student model to imitate a large, accurate teacher. The clever part is what the student copies. A normal label is hard — "this image is a cat, everything else is 0%". The teacher's full probability distribution is a soft label: "94% cat, 4% dog, 1% bird". That extra structure — the teacher quietly saying "this looks a bit like a dog too" — is the dark knowledge the student learns from.

To expose that structure you raise the softmax temperature. Higher temperature flattens the distribution so the small probabilities become visible and informative. The student is trained to match these softened teacher outputs (often plus the real labels). The result is a small model that punches well above its size — DistilBERT is 40% smaller than BERT while keeping about 97% of its accuracy.

Worked example — the distillation loss in PyTorch:

# The core of knowledge distillation: a student copies a teacher's
# SOFT labels (full probability distribution), not just the hard label.

import torch
import torch.nn.functional as F

T = 4.0   # temperature: higher T = softer, more informative targets

# Logits the teacher and student produce for one example, 5 classes.
teacher_logits = torch.tensor([[5.0, 2.0, 0.5, 0.1, -1.0]])
student_logits = torch.tensor([[4.2, 2.3, 0.4, 0.2, -0.8]])

# Soften both with temperature, then match them with KL-divergence.
teacher_soft = F.softmax(teacher_logits / T, dim=1)
student_soft = F.log_softmax(student_logits / T, dim=1)

distill_loss = F.kl_div(student_soft, teacher_soft, reduction="batchmean") * (T * T)

print(f"Teacher soft: {teacher_soft.round(decimals=3).tolist()[0]}")
print(f"Distill loss: {distill_loss.item():.4f}")

# Expected output (approximate):
#   Teacher soft: [0.63, 0.298, 0.205, ...]
#   Distill loss: 0.0009

The KL-divergence measures how far the student's softened output is from the teacher's. Multiplying by T*T keeps the gradient scale right when the temperature is high. Minimise this and the student's distribution slides toward the teacher's.

A linear layer is a weight matrix. A 1000×1000 matrix holds a million numbers. Low-rank factorization approximates that matrix as the product of two skinny matrices — say 1000×r times r×1000 for a small rank r. With r = 50 that is 1000×50 + 50×1000 = 100,000 numbers instead of 1,000,000 — a 10x reduction for that layer.

It works because real weight matrices are often low rank: their information lives in far fewer dimensions than their shape suggests, so a smaller rank captures almost all of it. The same idea powers LoRA, the dominant way to fine-tune large language models cheaply — you train only a tiny low-rank update instead of the full matrix. The cost is that too small a rank throws away real signal, so you pick the smallest r your accuracy can tolerate.

# Parameter count: full matrix vs a rank-r factorization
d_in, d_out, r = 1000, 1000, 50

full   = d_in * d_out              # 1,000,000 numbers
factor = d_in * r + r * d_out      # 100,000 numbers
print(f"full:   {full:,}")         # full:   1,000,000
print(f"rank-{r}: {factor:,}")     # rank-50: 100,000
print(f"compression: {full / factor:.0f}x")  # compression: 10x

There is no free lunch — every technique spends a little accuracy to buy size and speed. Choosing well means knowing your target. Deploying to a watch? You need maximum size and latency savings and can accept more accuracy loss. Serving a medical model? Guard accuracy and compress conservatively.

In practice you stack techniques: distill to a smaller architecture, prune it structurally, then quantize to INT8 — combinations reach 10–50x compression on edge devices. Crucially, some tasks live near an accuracy cliff: maths reasoning and code generation collapse under aggressive quantization, while text classification barely notices. Always benchmark on your task, not a generic one.

These four mistakes sink most first compression attempts: