Neural Networks Introduction

A neuron (also called a perceptron when it's on its own) is the smallest building block of a neural network. It takes some inputs and produces a single number. It does this in three tiny steps:

1. Weighted sum — multiply each input by its weight and add the results together.
2. Add a bias — add one extra number that shifts the total up or down.
3. Activation — pass the total through a function that decides the output.

The weights and bias are the neuron's knowledge. Everything a network learns ends up stored as weights and biases. Here is a single neuron written in plain Python — read each comment, then run it.

# A single neuron (a "perceptron") — plain Python, no libraries
# A neuron does 3 tiny steps:
#   1. weighted sum:  multiply each input by its weight and add them up
#   2. add a bias:    a number that shifts the result up or down
#   3. activation:    squash the result through a function (here: a step)

# The neuron's "knowledge" lives in these numbers.
inputs  = [1.0, 0.0, 1.0]      # 3 features going in (e.g. yes/no signals)
weights = [0.7, 0.3, 0.9]      # how much each input matters
bias   
...

The activation function is what makes a network powerful. It introduces non-linearity — a fancy way of saying "the output can bend and curve instead of being one straight line." Two activations cover almost everything a beginner needs:

• ReLU (Rectified Linear Unit): returns the input if it's positive, otherwise 0. It's fast and is the default choice for hidden layers.
• Sigmoid: squashes any number into the range 0 to 1 using an S-shaped curve. Perfect for an output that represents a probability ("how likely is this a cat?").

Both are just a couple of lines of plain Python — the only thing you need from the standard library is math.exp for sigmoid.

import math   # only the standard library — no numpy

# Activation functions decide what a neuron "fires".
# Without them, stacking neurons would just be one big straight line.

def relu(x):
    # ReLU = "Rectified Linear Unit": keep positives, zero out negatives
    return x if x > 0 else 0.0

def sigmoid(x):
    # Sigmoid squashes ANY number into the range 0..1 (an S-curve)
    return 1 / (1 + math.exp(-x))

# Try them on a few numbers so you can see the shape
for x in [-2.0, -0.5, 0.0, 0.5, 2
...

One neuron can only draw a single straight boundary, which is too weak for most real problems. The fix is to use many neurons arranged in layers:

• Input layer — your raw features (e.g. pixel values, sensor readings).
• Hidden layer(s) — neurons that each look for a different pattern in the inputs.
• Output layer — produces the final prediction.

The forward pass is simply running data through the network from left to right: every neuron in a layer computes weighted sum + bias + activation, and its outputs become the inputs to the next layer. Stack enough layers and the network can approximate almost any function — that's the whole magic.

A fresh network starts with random weights, so its first predictions are basically guesses. Training fixes that with a repeating loop:

1. Forward pass: run an example through the network to get a prediction.
2. Measure error: compare the prediction to the correct answer (the "loss").
3. Backpropagation: work out how much each weight contributed to the error.
4. Update: nudge every weight and bias a little in the direction that lowers the error.

That nudge size is controlled by the learning rate. Repeat this loop over thousands of examples and the weights slowly settle into values that make good predictions. You don't have to compute the gradients by hand — libraries like TensorFlow and PyTorch do it for you. Here's the same neuron idea written the professional way, plus a tiny Keras network, shown as a read-only reference.

# The SAME idea, written the way professionals do it.
# numpy does the weighted sum for a whole layer in one line.
import numpy as np

inputs  = np.array([1.0, 0.0, 1.0])
weights = np.array([0.7, 0.3, 0.9])
bias    = -1.0

z = np.dot(inputs, weights) + bias   # weighted sum + bias, vectorised
output = 1 / (1 + np.exp(-z))        # sigmoid activation
print(round(float(output), 3))       # 0.646

# Expected output:
# 0.646


# A whole network in a few lines with Keras (TensorFlow).
# This builds 2
...

These four mistakes trip up nearly everyone who builds their first network.

# ❌ hidden = weighted_sum            # linear — no power
# ✅ hidden = relu(weighted_sum)       # adds the curve the network needs

# hidden layers -> relu(z)            # gradient stays healthy
# output layer  -> sigmoid(z)         # 0..1 probability is fine here

# scale each feature to roughly 0..1
normalized = [(v - low) / (high - low) for v in feature]

# learning_rate = 10.0    # ❌ loss jumps around / becomes nan
learning_rate = 0.01      # ✅ steady, reliable improvement