Dimensionality Reduction

Squeeze data with hundreds of features down to a handful you can plot and model — using mean-centring, PCA, and t-SNE/UMAP — without throwing away the signal.

Learn Dimensionality Reduction in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🌍 Real-World Analogy: A Shadow on the Wall

Hold up your hand and shine a torch at it. The shadow on the wall is your 3D hand flattened to 2D. You lose depth, but if you turn your hand to a good angle, the shadow still tells you it is a hand — five fingers, a thumb, the shape is all there.

Dimensionality reduction is choosing the best angle to cast the shadow . Your data might live in 100 dimensions, but a well-chosen 2D shadow can keep the parts that matter — the clusters, the trends — and drop the rest. PCA picks the angle that keeps the shadow as spread out as possible (maximum variance), because a spread-out shadow preserves the most information. A summary of a long book keeps the plot and loses the filler; a good reduction keeps the structure and loses the noise.

1 The Curse of Dimensionality

A dimension is just a feature — one column in your dataset. Add more columns and you add more dimensions. That sounds harmless, but space gets enormous fast. To cover a 1D line with points spaced 0.1 apart you need 10 points; a 2D square needs 100; a 10-dimensional cube needs 10 billion. Your dataset of a few thousand rows is a tiny sprinkle in that vastness.

The painful consequence: in high dimensions, every point ends up roughly the same distance from every other point . Anything that relies on "near" versus "far" — k-nearest-neighbours, k-means, clustering, many distance kernels — starts to fall apart because "near" stops meaning anything. Models also need exponentially more data to fill the space, so they overfit.

2 PCA — Variance and Principal Components

PCA (Principal Component Analysis) is the workhorse of reduction. The idea in one line: find the direction in which your data is most spread out , call it the first principal component, then the next-most-spread direction at right angles to it, and so on.

Why chase spread? Because variance is information . A direction where every point has the same value tells you nothing; a direction where points spread far apart separates them. The first principal component is the single axis that captures the most variance — the best 1D "shadow" of your data. The second captures the most of what is left, and is perpendicular to the first.

The recipe is exactly the worked example below: mean-centre the data (so variance is measured from the centre), then project each point onto the chosen axis with a dot product. Project onto the top one or two components and you have reduced the data while keeping the most spread.

• Mean-centre: subtract each column's mean so the cloud sits at the origin.
• Project: take the dot product of each point with a unit axis to collapse it to one number.

Worked example. This is PCA's engine in plain Python — no libraries. Read every comment, then run it. It mean-centres a small 2D dataset and projects the points onto a single axis.

3 Explained Variance — How Many Components?

Doing the eigen-maths by hand is fine for two dimensions, but in practice you use scikit-learn . The number you care about most is the explained variance ratio : the fraction of total variance each principal component keeps. Add them up and you know how much information survives at each cut-off.

A common rule is to keep enough components to retain 90–95% of the cumulative variance. Read it straight off explained_variance_ratio_ . Run the worked example below: four correlated-plus-noise features collapse to two components that hold ~95% of the information.

4 t-SNE and UMAP — Reduction for Your Eyes

PCA is linear: it can only cast straight shadows. When clusters curl around each other, a straight shadow smears them together. t-SNE and UMAP are non-linear methods built for one job — making a 2D or 3D picture in which similar points sit close together so clusters become visible.

t-SNE focuses on local structure : it tries to keep each point's nearest neighbours nearby. The perplexity setting controls roughly how many neighbours each point balances (typical 5–50). UMAP does a similar job, usually runs faster, and tends to preserve more of the global layout.

5 Feature Selection vs Feature Extraction

There are two different ways to end up with fewer features, and people mix them up constantly:

Keep a subset of the original columns and drop the rest. The survivors are still your real, named features, so the result stays interpretable . Examples: drop low-variance columns, drop one of two highly correlated columns, or use model importances.

Build new features by combining the originals. PCA, t-SNE, UMAP, and autoencoders all do this. You usually pack more information into fewer features, but the new features (like "PC1") are harder to read .

Rule of thumb: if you need to explain the result to a human or a regulator, prefer selection. If you only need a compact, accurate input for a model or a plot, extraction usually wins.

🧭 When (and When Not) to Reduce Dimensions

• You want to visualise high-dimensional data in 2D or 3D (t-SNE / UMAP).
• Training is slow or models overfit because there are too many features.
• Features are highly correlated and you want to decorrelate them (PCA).
• You want to strip noise so the signal is cleaner.
• You only have a handful of features already — reduction may just lose information.
• Interpretability is essential — PCA components are hard to explain.
• A tree-based model (random forest, gradient boosting) is doing fine; these handle many features and irrelevant columns well, so PCA often hurts more than it helps.

Now you try. Fill in the blanks marked ___ . Mean-centring is step 1 of every PCA.

Your turn again. Project the centred points onto one axis with a dot product, then measure how much variance survived.

6 Common Errors (And How to Fix Them)

These four mistakes bite almost everyone learning dimensionality reduction:

A feature in thousands (salary) drowns out a feature in fractions (a ratio) just because its numbers are bigger, so PC1 ends up being "salary" by accident.

❌ Reading meaning into t-SNE distances and cluster sizes

"These two clusters are far apart, so they're very different" and "this cluster is huge, so it has more points" are both wrong . t-SNE distorts gaps and inflates dense regions on purpose.

✅ Fix: read t-SNE for which points group together only. Try several perplexity values, and confirm any real claim with PCA or the raw distances.

Jumping straight to n_components=2 because it plots nicely can throw away most of your signal if those two components only explain, say, 40% of the variance.

Calling fit (or fit_transform ) on the whole dataset before splitting lets the scaler and PCA peek at the test data. Your scores look great, then collapse in production.

✅ Fix: fit on train only, then transform test — a Pipeline does this safely inside cross-validation.

📋 Quick Reference

Method

Type

Use for output in a model?

Best for

PCA

Linear, extraction

Yes

Compress correlated features, decorrelate, denoise

t-SNE

Non-linear

No — viz only

Seeing local clusters in 2D/3D

UMAP

Faster t-SNE with better global layout

Feature selection

Keeps originals

Interpretable results, dropping redundant columns

explained_variance_ratio_

Diagnostic

—

Choosing how many components to keep (90–95%)

❓ Frequently Asked Questions

🎯 Mini-Challenge: Selection vs Extraction by Hand

Time to fly with less scaffolding. The starter has a tiny 3-feature dataset and only a comment outline — write the logic yourself. One feature is a near-constant (it carries no signal); two carry the real pattern.

Lesson 16 complete — you can now shrink data without losing the plot!

You understand the curse of dimensionality, you can mean-centre and project data by hand, run PCA and read its explained variance, reach for t-SNE or UMAP to see clusters, and tell feature selection from feature extraction. You also know the four traps: not scaling, over-reading t-SNE, keeping too few components, and leaking test data.

🚀 Up next: Ensemble Methods — combine many models into one that beats them all.

Practice quiz

Continue this course

• Previous: Unsupervised Learning
• Next: Ensemble Methods