Dimensionality Reduction with PCA

Reviewed & published by Brayan K

Squash many correlated features into a handful of new axes that keep most of the information. Learn how PCA finds directions of maximum variance and how to read the explained variance ratio.

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• Why variance is where the information lives
• What principal components actually are
• How to read the explained variance ratio
• How to choose how many components to keep
• Why scaling first is mandatory for PCA
• When to use PCA — and when not to

🎯 Real-World Analogy: The Best Camera Angle

To photograph a 3D sculpture as a single 2D photo, you choose the angle that shows the most detail — you'd never shoot it edge-on, where everything collapses into a thin line. PCA does the same with data: it finds the viewing angle (direction) along which the points spread out the most, then flattens onto it while losing as little as possible.

That best angle is the first principal component. The next-best angle, at right angles to it, is the second, and so on.

1 Variance Is Information

PCA rests on one idea: the directions where your data spreads out the most carry the most information. A direction where all the points sit on top of each other tells you nothing; a direction where they fan out tells you a lot. That spread is the variance.

So PCA's job is to find new axes — principal components — ordered by how much variance each one captures. PC1 captures the most, PC2 the next-most (and is orthogonal to PC1), and so on. Each component is a linear combination (weighted blend) of your original features.

PCA in plain English:

• Center the data (subtract each feature's mean).
• Find the direction of greatest variance — that's PC1.
• Find the next direction of greatest variance at right angles — PC2.
• Keep the top few components and drop the rest.

2 Measuring Variance Along a Direction

Let's make it concrete. We project the data onto candidate directions and measure the spread of the resulting 1D shadow. The direction with the most variance is closest to the true first component.

# PCA is all about VARIANCE: the direction where data spreads most.
# Here we measure variance along two candidate directions by hand (numpy).
import numpy as np

# Data that mostly spreads along the diagonal (x grows with y).
X = np.array([[2.0, 2.1], [3.0, 3.0], [4.0, 4.2], [5.0, 4.9], [6.0, 6.1]])
X = X - X.mean(axis=0)              # PCA always centers the data first

# Project the data onto a unit direction, then measure the spread (variance).
def variance_along(direction):
    direction = direction / np.linalg.norm(direction)   # make it length 1
    projected = X @ direction                            # 1D shadow of each point
    return float(np.var(projected))

print(round(variance_along(np.array([1.0, 0.0])), 2))   # along x only
print(round(variance_along(np.array([0.0, 1.0])), 2))   # along y only
print(round(variance_along(np.array([1.0, 1.0])), 2))   # along the diagonal

# The diagonal captures the MOST variance -> it is close to PC1.

# Expected output:
# 2.0
# 2.06
# 4.05

3 Explained Variance Ratio

Each component captures a fraction of the data's total variance — its explained variance ratio. These fractions are sorted high to low and they add up to 1.0 across all components. To decide how many components to keep, add them from the top until you hit a target like 90% or 95%.

For example, if PC1 keeps 0.85 and PC2 keeps 0.10, then two components keep 0.85 + 0.10 = 0.95 — 95% of the variance with half the axes you might have started with.

🌍 Worked Example: The Real Tool (scikit-learn)

Here's PCA with scikit-learn's PCA, after scaling. Study it (it isn't runnable in the in-browser sandbox). Notice fit_transform turns 4 features into 2 components, and explained_variance_ratio_ tells you how much each one kept.

# WORKED EXAMPLE — what the real tool looks like (scikit-learn).
# You don't run this here; study it so the library code feels familiar.
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# 4 correlated features -> compress to 2 components.
X = [[2, 4, 1, 9], [3, 6, 2, 8], [4, 8, 1, 7], [5, 9, 3, 6]]

X_scaled = StandardScaler().fit_transform(X)   # ALWAYS scale before PCA

pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X_scaled)        # 4 features -> 2 components

print("shape:", X_reduced.shape)               # (4, 2)
print("explained variance ratio:", pca.explained_variance_ratio_.round(2))
print("total kept:", round(pca.explained_variance_ratio_.sum(), 2))

# Expected output (values illustrative):
# shape: (4, 2)
# explained variance ratio: [0.9x 0.0x]
# total kept: ~0.99

The scaler comes before PCA in the pipeline — skipping it lets the largest-range feature hijack the first component.

4 When to Use PCA (and the Catch)

Reach for PCA when you have many features, especially correlated ones. It speeds up training, reduces noise, fights the "curse of dimensionality", and lets you plot high-dimensional data in 2D or 3D.

🎯 Your Turn 1: Finish variance_along()

Fill in the blanks so the helper projects the data onto a direction and returns the variance of the shadow. The expected output is in the comments.

# 🎯 YOUR TURN — finish the variance_along() helper.
# Project the centered data onto a unit direction and measure the spread.
import numpy as np

X = np.array([[1.0, 1.0], [2.0, 2.0], [3.0, 3.0], [4.0, 4.0]])
X = X - X.mean(axis=0)            # center first

def variance_along(direction):
    direction = direction / np.linalg.norm(direction)
    projected = X @ ___          # 👉 project onto 'direction' with the @ operator
    return float(np.var(___))    # 👉 variance of the 'projected' shadow

print(round(variance_along(np.array([1.0, 1.0])), 2))   # 👉 should be 2.5
print(round(variance_along(np.array([1.0, -1.0])), 2))  # 👉 should be 0.0

# ✅ Expected output:
# 2.5
# 0.0

🎯 Your Turn 2: How Many Components?

Add explained-variance ratios until you reach a target. Fill in the blank so the function returns the number of components needed for 90% variance.

# 🎯 YOUR TURN — decide how many components to keep for 90% variance.
# Add explained-variance ratios from strongest down until you reach 0.90.

ratios = [0.70, 0.18, 0.07, 0.05]   # one per component, strongest first

def components_for(target):
    total = 0.0
    for i, r in enumerate(ratios):
        total += r
        if total >= ___:        # 👉 compare the running total to 'target'
            return i + 1        # 👉 number of components used so far
    return len(ratios)

print(components_for(0.90))     # 👉 should be 3 (0.70 + 0.18 + 0.07 = 0.95)

# ✅ Expected output:
# 3

🎯 Mini-Challenge: Cumulative Variance

Given each component's explained variance, print the running cumulative total and how many components reach 95%. Only a comment outline is provided.

# 🎯 MINI-CHALLENGE: report cumulative explained variance
# You are given the explained-variance ratio of each component.
#
# 1. ratios = [0.62, 0.23, 0.10, 0.05]
# 2. Walk through them and keep a running CUMULATIVE total
# 3. print() the cumulative total after each component (rounded to 2)
# 4. print() how many components are needed to reach at least 0.95
#
# ✅ Expected cumulative: 0.62, 0.85, 0.95, 1.0  -> 3 components reach 0.95

# your code here

5 Common Errors (And How to Fix Them)

These are the most common PCA mistakes. Watch for them.

❌ Running PCA on unscaled data

The largest-range feature dominates the first component for no real reason.

# ❌ salary (in dollars) hijacks PC1
PCA(n_components=2).fit(X_raw)

PCA(n_components=2).fit(StandardScaler().fit_transform(X))

❌ Fitting the scaler/PCA on the test set

Fitting PCA on all data before splitting leaks test information into training — a form of data leakage.

PCA().fit(X_all)   # ❌ then split -> leakage

✅ Fix: fit on train only, transform the test set:

pca.fit(X_train); pca.transform(X_test)

❌ Using PCA when you need interpretable features

Components blend many features, so you can no longer explain a decision in plain terms.

# ❌ "approved because PC2 was high" means nothing to a regulator

✅ Fix: keep the original features (or use feature selection) when explanation matters:

model.fit(X_original, y)   # interpretable columns

📋 Quick Reference

🎉 Lesson Complete!

You now understand how PCA finds the directions of greatest variance, what principal components are, how to read the explained variance ratio, and why you must scale first.

Practice quiz

Continue this course

• Previous: K-Means & Clustering
• Next: Unsupervised Learning — Finding structure in unlabelled data — clustering, association and density estimation
• Quick reference: AI & Machine Learning cheat sheet
• From the blog: Data Preprocessing for Machine Learning

Frequently asked questions

What is PCA in plain English?

Principal Component Analysis takes data with many features and finds a smaller set of new directions (principal components) that capture most of the information. It's like photographing a 3D object from the one angle that shows the most detail. You go from many columns down to a few, while throwing away as little signal as possible.

How does PCA choose its directions?

It looks for the direction along which the data spreads out the most — that's the first principal component, because spread (variance) is where the information is. The second component is the direction of next-most variance that is at right angles (orthogonal) to the first, and so on. Each component is a weighted mix of your original features.

What is the explained variance ratio and how do I use it?

Each component captures some fraction of the data's total variance — that fraction is its explained variance ratio. You add them up from the strongest component down and keep just enough components to reach a target, often 90% or 95%. For example, if PC1 keeps 0.85 and PC2 keeps 0.10, two components keep 0.95 of the variance.

Why do I have to scale features before PCA?

PCA is driven entirely by variance. A feature measured in large units (like salary in dollars) has a huge raw variance compared with a feature in small units (like years of experience), so it would dominate the first component for no good reason. Standardising every feature to comparable units lets PCA judge directions fairly.

When should I use PCA, and when should I avoid it?

Use PCA when you have many features, especially correlated ones, and you want to speed up a model, reduce noise, or visualise data in 2D or 3D. Avoid it when you need the original features to stay interpretable — for a doctor or a regulator, 'principal component 3' means nothing, whereas 'blood pressure' does.

Does PCA need labels?

No. PCA is unsupervised — it only looks at the features and their variance, never the target label. That makes it a preprocessing step you can apply before either a supervised model or another unsupervised method like clustering.