Support Vector Machines

Reviewed & published by Brayan K

Learn how an SVM finds the widest possible gap between classes, why only a few "support vector" points matter, and how the kernel trick bends straight lines into curves.

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• What a hyperplane and a margin really are
• Why support vectors are the only points that matter
• How the kernel trick handles non-linear data
• The difference between RBF and polynomial kernels
• How C trades margin width against training errors
• How gamma controls the reach of each point in RBF

🎯 Real-World Analogy: The Widest Street

Imagine two neighbourhoods you must separate with a road. You could draw the road hugging one side, but the safest choice is the widest street that keeps the most distance from the nearest house on either side. That central road is the SVM's decision boundary, and the empty space on each side is the margin.

The houses closest to the road — the ones that determine how wide it can be — are the support vectors. Move a faraway house and nothing changes; move a support vector and the whole road shifts.

1 Hyperplanes, Margins & Support Vectors

A hyperplane is just the generalisation of a line (2D) or a plane (3D) to any number of dimensions — it's the flat surface that separates the classes. Many hyperplanes can split the same data, so the SVM adds a rule: pick the one with the maximum margin.

The margin is the distance from the hyperplane to the nearest training point of each class. The few points that sit exactly on the edge of that margin are the support vectors. Remarkably, they are the only points that define the boundary — delete the rest and the SVM is unchanged.

The SVM idea, in plain English:

• Of all the boundaries that separate the classes, consider the gap each one leaves.
• Pick the boundary that leaves the widest gap (margin) to the nearest points.
• Those nearest points are the support vectors.
• A wide margin tends to generalise better to unseen data.

2 Measuring the Margin

Let's make the margin concrete. With one feature, a boundary is just a threshold. The margin is the distance from that threshold to the closest point of either class. Run this and watch which threshold gives the widest gap.

# How an SVM scores a boundary: it maximises the MARGIN.
# The margin is the distance from the decision line to the nearest point.
# A wider margin usually means better generalisation to new data.
import numpy as np

# Two classes in 1D for intuition. Class A around 2, class B around 8.
class_a = np.array([1.0, 2.0, 3.0])
class_b = np.array([7.0, 8.0, 9.0])

# A candidate boundary sits at some threshold t. The margin is the
# distance from t to the nearest point on either side.
def margin(t):
    nearest_a = np.min(np.abs(class_a - t))   # closest A point
    nearest_b = np.min(np.abs(class_b - t))   # closest B point
    return min(nearest_a, nearest_b)          # margin = the smaller gap

for t in [4.0, 5.0, 6.0]:
    print("boundary at", t, "-> margin", round(margin(t), 2))

# The SVM would choose t = 5.0: it sits exactly halfway, giving the
# widest margin of 2.0. Points at 3 and 7 are the SUPPORT VECTORS.

# Expected output:
# boundary at 4.0 -> margin 1.0
# boundary at 5.0 -> margin 2.0
# boundary at 6.0 -> margin 1.0

3 The Kernel Trick

Real data is rarely separable by a straight line. The kernel trick lets a linear SVM draw curved boundaries. A kernel function measures how similar two points are as if they had been mapped into a much higher-dimensional space — without ever building that space.

• Linear kernel: a plain straight boundary; best when the data is already separable and you have many features.
• Polynomial kernel: boundaries shaped like curves of a chosen degree.
• RBF (Gaussian) kernel: smooth, flexible, closed curves; the common default for non-linear problems.

🌍 Worked Example: The Real Tool (scikit-learn)

In practice you call a library. Here's an RBF SVM with scikit-learn's SVC, wrapped in a scaler because SVMs are scale-sensitive. Study it (it isn't runnable in the in-browser sandbox) and notice how the concepts map: kernel="rbf" is the kernel trick, C trades margin width against errors, and gamma sets each point's reach.

# WORKED EXAMPLE — what the real tool looks like (scikit-learn).
# You don't run this here; study it so the library code feels familiar.
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Features: [hours_studied, hours_slept]   Label: passed? (0 = no, 1 = yes)
X = [[1, 4], [2, 5], [8, 7], [9, 8], [3, 4], [7, 6]]
y = [0, 0, 1, 1, 0, 1]

# ALWAYS scale before an SVM (distances are scale-sensitive).
# kernel="rbf" handles curved boundaries; C and gamma control the fit.
model = make_pipeline(
    StandardScaler(),
    SVC(kernel="rbf", C=1.0, gamma="scale")
)
model.fit(X, y)                      # learns the support vectors here

print(model.predict([[6, 6]]))       # predict a new student
print(model.named_steps["svc"].n_support_)   # support vectors per class

# Expected output:
# [1]
# [1 1]

The expected output is shown in the comments. n_support_ tells you how many support vectors each class contributed — often just a small fraction of the data.

4 Tuning C and Gamma

Two knobs control how an RBF SVM fits. C is the regularisation strength: small C allows a wider margin and tolerates some misclassified points; large C insists on getting the training points right, narrowing the margin and risking overfitting. Gamma sets how far the influence of a single point reaches: small gamma gives smooth, broad boundaries; large gamma makes each point only matter locally, producing wiggly boundaries that can overfit.

🎯 Your Turn 1: Finish the Margin Function

Fill in the blanks so margin() returns the distance to the nearest point of either class. The expected output is in the comments so you can self-check.

# 🎯 YOUR TURN — finish the margin() function
# The margin is the distance from the boundary t to the NEAREST point.
import numpy as np

points_a = np.array([1.0, 2.0, 3.0])
points_b = np.array([7.0, 8.0, 9.0])

def margin(t):
    nearest_a = np.min(np.abs(points_a - t))
    nearest_b = ___                       # 👉 same idea for class B
    return ___                            # 👉 return the SMALLER of the two

print(round(margin(5.0), 2))              # 👉 should be 2.0 (the widest)
print(round(margin(4.0), 2))             # 👉 should be 1.0

# ✅ Expected output:
# 2.0
# 1.0

🎯 Your Turn 2: Choose C Wisely

Decide whether to use a small or large C based on how noisy the data is. Fill in the two blanks so the recommendation matches the rule of thumb.

# 🎯 YOUR TURN — pick C for a noisy vs clean dataset.
# Rule of thumb: SMALL C = wider margin, tolerates errors (more regularised).
#                LARGE C = punishes errors, narrow margin (can overfit).

def recommend_C(data_is_noisy):
    if data_is_noisy:
        return ___        # 👉 return "small C" (let the margin be forgiving)
    else:
        return ___        # 👉 return "large C" (clean data, fit tightly)

print(recommend_C(True))
print(recommend_C(False))

# ✅ Expected output:
# small C
# large C

🎯 Mini-Challenge: Maximum-Margin Boundary

Now do it with the support faded. Given two 1D classes, scan candidate thresholds and print which one gives the widest margin. Only a comment outline is provided.

# 🎯 MINI-CHALLENGE: find the maximum-margin boundary by hand
# You have two 1D classes. Scan candidate thresholds and report the best.
#
# 1. class_a = [2, 3, 4]   class_b = [8, 9, 10]
# 2. For each t in range, margin(t) = min distance to nearest point of EITHER class
# 3. Try thresholds t = 5, 6, 7
# 4. print() each margin, then print the t with the LARGEST margin
#
# ✅ Expected: t=6 gives the widest margin (2.0); support vectors are 4 and 8

# your code here

5 Common Errors (And How to Fix Them)

These traps catch almost everyone working with SVMs. Watch for them.

❌ Forgetting to scale the features

An unscaled large-range feature dominates the distance calculation and wrecks the boundary.

# ❌ No scaling — income (in dollars) swamps age (in years)
SVC(kernel="rbf").fit(X_raw, y)

✅ Fix: standardise inside a pipeline:

make_pipeline(StandardScaler(), SVC(kernel="rbf")).fit(X, y)

❌ Cranking C and gamma to the maximum

Huge C and huge gamma together memorise the training set — perfect train score, poor test score.

SVC(C=100000, gamma=1000)   # overfits badly

✅ Fix: tune both with cross-validation:

GridSearchCV(SVC(), {"C": [0.1, 1, 10], "gamma": [0.01, 0.1, 1]})

❌ Using an SVM on a huge dataset

SVM training time grows steeply with the number of rows; on millions of rows it can be impractically slow.

# ❌ 5 million rows -> SVC may take hours
SVC(kernel="rbf").fit(X_huge, y_huge)

✅ Fix: use a linear model or tree ensemble for very large data:

LinearSVC().fit(X_huge, y_huge)   # or LogisticRegression / RandomForest

📋 Quick Reference

🎉 Lesson Complete!

You now understand how an SVM finds the maximum-margin hyperplane, why only the support vectors matter, how the kernel trick bends the boundary, and how C and gamma control the fit.

Practice quiz

Continue this course

• Previous: Model Deployment
• Next: K-Means & Clustering — Unsupervised grouping with k-means, choosing k, DBSCAN and hierarchical
• Quick reference: AI & Machine Learning cheat sheet

Frequently asked questions

What is a Support Vector Machine in simple terms?

An SVM is a classifier that draws the best possible boundary between two classes. Out of all the lines (or planes) that could separate the data, it picks the one with the widest gap — the margin — to the nearest points of each class. That wide buffer tends to generalise well to new data.

What are support vectors and why do they matter?

Support vectors are the handful of training points that sit right on the edge of the margin. They are the only points that actually define the boundary — you could delete every other point and the SVM would learn the exact same line. This is why SVMs can be memory-efficient at prediction time.

What is the kernel trick?

Many datasets cannot be split by a straight line. The kernel trick computes how similar pairs of points are as if they had been lifted into a much higher-dimensional space, without ever building that space explicitly. A linear boundary in that hidden space looks like a curved boundary in the original one. The RBF and polynomial kernels are the most common.

What is the difference between C and gamma?

C controls the margin/error trade-off for every SVM: small C gives a wider, more forgiving margin (more regularisation), large C punishes misclassifications harder and can overfit. Gamma is specific to the RBF kernel: it sets how far the influence of a single point reaches. Small gamma = smooth, broad boundary; large gamma = tight, wiggly boundary that can overfit.

Do I need to scale features for an SVM?

Yes, almost always. SVMs measure distances and dot products between points, so a feature with a large numeric range (like income in dollars) will dominate a small-range feature (like age in years). Standardising every feature to a similar scale is a required preprocessing step for reliable SVM results.

When should I pick an SVM over other models?

SVMs are a great fit for small-to-medium datasets, especially when there are many features (even more features than rows) and a fairly clear separation. They struggle on very large datasets because training time grows quickly with the number of rows — there a tree ensemble or logistic regression is usually faster.

Related lessons

• Classification Basics