Advanced Neural Network Techniques

An activation function sits after each layer and decides what the neuron passes on. Without one, stacking layers would just be one big linear function — the network could not learn curves or anything interesting. The activation adds the non-linearity that lets a network model complex patterns.

• ReLU (Rectified Linear Unit): outputs the value if positive, else 0. Fast and the most common default.
• LeakyReLU: like ReLU, but negatives leak through with a small slope so neurons cannot get permanently stuck at 0.
• GELU (Gaussian Error Linear Unit): a smooth curve used in transformers and most modern large language models.

Here is each one built from scratch in plain Python so you can watch what they do. Run it:

# Activation functions decide what a neuron passes on.
# We build them from scratch so you can SEE what they do —
# no libraries, just lists and simple math.

def relu(values):
    # ReLU: keep positives, clamp negatives to 0
    return [max(0.0, x) for x in values]

def leaky_relu(values, slope=0.01):
    # LeakyReLU: like ReLU, but negatives leak through a little
    # so the neuron never fully "dies"
    return [x if x > 0 else slope * x for x in values]

scores = [3.0, -2.0, 0.5, -10.0]

pri
...

Notice ReLU flattens every negative to 0.0, while LeakyReLU keeps a faint negative signal (-2.0 becomes -0.02). That tiny leak is what keeps the neuron alive during training.

from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Dense(64, activation="relu"),    # ReLU hidden layer
    layers.Dense(64, activation="gelu"),    # GELU hidden layer
    layers.Dense(10, activation="softmax"), # softmax OUTPUT for 10 classes
])

# Expected output: a model whose hidden layers use ReLU and GELU,
# and whose output layer turns scores into class probabilities.

Before training starts, every weight needs a starting value. Set them all to zero and every neuron learns the same thing (useless). Set them too large and the signal explodes; too small and it vanishes. Smart initialization scales the random starting weights by the layer size so signals stay a healthy size on the very first forward pass.

• Xavier (Glorot): tuned for symmetric activations like tanh and sigmoid.
• He: scales variance up by 2 to account for ReLU discarding the negative half — use it with ReLU.

from tensorflow.keras import layers

# He init pairs with ReLU
layers.Dense(64, activation="relu",
             kernel_initializer="he_normal")

# Xavier/Glorot init pairs with tanh
layers.Dense(64, activation="tanh",
             kernel_initializer="glorot_uniform")

# Expected output: layers whose initial weights are scaled to the
# right size for their activation, so early training stays stable.

As data flows through a deep network, the scale of values can drift wildly between layers, which slows training. Normalization re-centers and re-scales these values so each layer sees inputs in a stable range — like a cooling system keeping the engine from overheating.

• BatchNorm: normalizes each feature across the examples in a mini-batch. Great for CNNs; needs a decent batch size and behaves differently at inference.
• LayerNorm: normalizes across the features of a single example. Independent of batch size, identical in training and inference — the default in transformers.

from tensorflow.keras import layers

# Typical CNN pattern: Dense -> BatchNorm -> activation
layers.Dense(64, use_bias=False)
layers.BatchNormalization()
layers.Activation("relu")

# Transformer pattern uses LayerNorm instead
layers.LayerNormalization()

# Expected output: activations kept in a stable range layer to layer,
# letting you train deeper networks faster.

Overfitting is when a model memorizes the training data instead of learning the general pattern — it scores great on data it has seen and badly on anything new. Regularization is the set of tools that prevent this.

• Dropout: randomly switch off a fraction of neurons each training step, so the network cannot lean on any single unit.
• L2 (weight decay): add a penalty for large weights, nudging the model toward simpler, smoother solutions.
• Early stopping: watch validation loss and stop training the moment it stops improving.

from tensorflow import keras
from tensorflow.keras import layers, regularizers

model = keras.Sequential([
    layers.Dense(128, activation="relu",
                 kernel_regularizer=regularizers.l2(1e-4)),  # L2
    layers.Dropout(0.3),                                     # drop 30%
    layers.Dense(10, activation="softmax"),
])

early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss", patience=3, restore_best_weights=True
)
# model.fit(..., callbacks=[early_stop])

# Expected output: training halts automatically once validation loss
# stops improving, and the best (not the last) weights are kept.

The optimizer is the rule that turns gradients into weight updates. Plain SGD takes a step straight downhill. Momentum adds memory of the previous direction so the optimizer keeps rolling through small bumps and noisy gradients — like a ball gaining speed downhill. Adam goes further, giving every weight its own adaptive step size.

Below is a single momentum update step in plain Python. The formula is just two lines: velocity = beta * velocity - lr * gradient, then weight = weight + velocity. Run it:

# One step of SGD WITH MOMENTUM, in plain Python.
# Momentum remembers the previous update direction so the
# optimizer keeps rolling and smooths out noisy gradients.
#
#   velocity = beta * velocity - lr * gradient
#   weight   = weight + velocity

lr = 0.1        # learning rate (step size)
beta = 0.9      # momentum factor (how much past direction to keep)

weight = 2.0    # the parameter we are training
velocity = 0.0  # momentum starts at rest
gradient = 0.5  # slope of the loss w.r.t. weig
...

from tensorflow import keras

# SGD with momentum
sgd = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)

# Adam — adaptive per-parameter step sizes (the safe default)
adam = keras.optimizers.Adam(learning_rate=1e-3)

model.compile(optimizer=adam, loss="categorical_crossentropy")

# Expected output: a compiled model that updates its weights with Adam,
# adapting the step size for every parameter automatically.

The learning rate is the size of each step. A fixed rate is a compromise: too big and the model bounces around the target; too small and training crawls. A schedule changes the rate over time — big early to cover ground, small later to settle precisely. It is the gearbox of training.

• Step decay: drop the rate by a factor every few epochs.
• Cosine annealing: smoothly curve the rate down to near zero.
• Warm-up: start tiny and ramp up over the first few hundred steps so an untrained net is not destabilized.

from tensorflow import keras

lr_schedule = keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=1e-3,
    decay_steps=10000,   # curve down to ~0 over 10k steps
)

opt = keras.optimizers.Adam(learning_rate=lr_schedule)

# Expected output: the learning rate starts at 1e-3 and smoothly
# decreases toward 0 as training proceeds, helping the model settle.

These five mistakes trip up almost everyone at first. Here is how to spot and fix each one: