Residual Networks & DenseNets

You'd expect a deeper network to be at least as good as a shallow one — worst case, the extra layers could just copy their input and do no harm. In practice, before 2015, the opposite happened. Stack enough plain layers and accuracy got worse, on the training set as well as the test set. A famous result showed a 56-layer network beaten by a 20-layer one.

This is the degradation problem, and it is not overfitting (overfitting would mean great training accuracy, poor test accuracy). It's an optimisation problem: as a gradient travels backwards through dozens of layers during training, it gets multiplied at every step. Multiply by small numbers enough times and it shrinks to almost nothing — the vanishing-gradient problem — so the early layers receive almost no signal telling them how to improve.

ResNet's idea is almost embarrassingly simple. Instead of asking a block to produce the full desired output H(x), ask it to produce only the residual — the change — and then add the original input back:

output = F(x) + x        # F is the block; x arrives via the skip connection
# rearranged: F(x) = output - x   ->  F learns the "residual" (the change)

The + x term is the skip connection (also called a shortcut or identity connection). It costs no extra parameters — it's just addition. Why does it help so much? Because if the best thing a block can do is leave its input alone (an identity mapping), it just needs to drive F(x) towards zero. Pushing weights towards zero is easy; learning to perfectly reproduce the input with raw weight layers is hard. The skip guarantees deeper networks can never be worse than shallower ones — the extra blocks can always fall back to doing nothing.

The example below builds one residual block in plain Python so you can see F(x) + x directly:

# A residual block in plain Python: output = F(x) + x
# No libraries needed — we use small lists and simple functions
# so you can watch the skip connection do its job.

def relu(v):
    # ReLU keeps positives, zeroes out negatives
    return [max(0.0, n) for n in v]

def scale(v, factor):
    # Stand-in for "a weight layer" — multiply every number
    return [n * factor for n in v]

def F(x):
    # The block's learned transformation (two tiny "layers")
    h = relu(scale(x, 0.5))   # layer 1 + 
...

Here's the heart of why ResNet works. When the training signal flows backwards, a plain layer multiplies the gradient by some factor. Across many layers those factors compound — and if they're below 1, the gradient vanishes. A skip connection changes the maths: because output = F(x) + x, the gradient gets a guaranteed "+1" path at every block, so it can never fully die.

The runnable example below contrasts the two. The plain column shrinks towards zero as depth grows; the residual column stays healthy:

# Why skip connections rescue deep networks: GRADIENT FLOW.
# When gradients travel back through N layers, a PLAIN net multiplies
# them by each layer's factor (they can shrink to ~0 = "vanishing").
# A residual net ADDS 1 at every block, so the signal never dies.

layer_factor = 0.5   # each plain layer weakens the gradient (factor < 1)
depths = [1, 5, 10, 20, 50]

print(f"{'depth':>6} {'plain':>14} {'residual':>12}")
for n in depths:
    plain = layer_factor ** n          # 0.5 * 0.5 * ... (n 
...

A 3x3 convolution on many channels is expensive. To build really deep ResNets (50, 101, 152 layers) without the cost exploding, ResNet uses a bottleneck block: a three-step sandwich.

1. 1x1 conv — squeeze: shrink the channel count (e.g. 256 → 64). Cheap.
2. 3x3 conv — work: do the real spatial processing on the small 64-channel tensor.
3. 1x1 conv — expand: grow the channels back (64 → 256) so the skip can add cleanly.

The narrow middle is the "bottleneck". By doing the heavy 3x3 work on far fewer channels, the block delivers similar power for a fraction of the computation — which is exactly how ResNet-50 stays trainable on ordinary hardware. The skip connection still wraps the whole sandwich: output = bottleneck(x) + x.

DenseNet (2017) pushed the skip idea to its limit. Where ResNet adds the input (F(x) + x), DenseNet concatenates: inside a dense block, every layer receives the stacked feature maps of all earlier layers as its input. Layer 4 sees the outputs of layers 1, 2, and 3 bundled together.

Each layer contributes only a small fixed number of new feature maps — the growth rate k (often 12 to 32). Because every layer can reach back to all earlier features, the layers themselves can be narrow, which keeps the total parameter count surprisingly low. Concatenation also gives every layer a direct line to the loss, so gradients flow even more freely than in ResNet.

# Inside a dense block (growth rate k), channels grow layer by layer:
# layer 1 input: 16          output adds k -> running total 16 + k
# layer 2 input: 16 + k      output adds k -> running total 16 + 2k
# layer 3 input: 16 + 2k     output adds k -> running total 16 + 3k
# ResNet would keep the channel count fixed by ADDING instead.

Because channels keep growing, DenseNet inserts transition layers (a 1x1 conv plus pooling) between dense blocks to compress the feature maps back down and keep memory in check.

You almost never train a ResNet from scratch. A ResNet trained on ImageNet (1.2 million images, 1000 classes) has already learned reusable visual features — edges, textures, shapes, object parts. Transfer learning reuses those learned features: load the pretrained weights, swap out the final classification layer for your own classes, and fine-tune. The two read-only examples below run locally with torchvision installed (pip install torch torchvision).

Load a pretrained model and classify an image:

# Loading a PRETRAINED ResNet-50 with torchvision (run locally).
# It already knows 1000 ImageNet classes — no training needed.
import torch
from torchvision.models import resnet50, ResNet50_Weights

weights = ResNet50_Weights.DEFAULT          # best available ImageNet weights
model = resnet50(weights=weights)
model.eval()                                 # inference mode (no dropout/BN updates)

preprocess = weights.transforms()            # the EXACT transforms it was trained with

from PIL import Image
img = Image.open("dog.jpg")
batch = preprocess(img).unsqueeze(0)         # shape: [1, 3, 224, 224]

with torch.no_grad():
    logits = model(batch)
probs = logits.softmax(dim=1)
top_p, top_id = probs.max(dim=1)

label = weights.meta["categories"][top_id.item()]
print(f"{label}: {top_p.item():.1%}")

# Expected output:
# Labrador retriever: 87.3%

Adapt it to your own classes by replacing only the final layer:

# TRANSFER LEARNING: reuse a pretrained ResNet on YOUR classes.
# Freeze the learned features, replace only the final layer.
import torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

model = resnet50(weights=ResNet50_Weights.DEFAULT)

# 1) Freeze every pretrained layer so training won't disturb them
for param in model.parameters():
    param.requires_grad = False

# 2) Replace the 1000-class head with one for YOUR task (say 3 classes)
num_features = model.fc.in_features          # 2048 for ResNet-50
model.fc = nn.Linear(num_features, 3)        # only THIS layer will train

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable parameters: {trainable}")  # tiny vs. 25M total

# Expected output:
# Trainable parameters: 6147