Ensemble Methods

By the end of this lesson you'll be able to combine several models into one stronger predictor — and explain exactly why a crowd of models beats the best individual model.

Learn Ensemble Methods in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🧠 Real-World Analogy: A Panel of Experts

Imagine a tricky medical case. You don't bet everything on one doctor — you convene a panel of specialists . The cardiologist, the radiologist, and the surgeon each see the problem from a different angle. Individually each can be wrong; but when most of the panel agrees on a diagnosis, you trust the consensus far more than any single voice.

An ensemble is exactly that panel, made of machine-learning models. Each model is an "expert" with its own blind spots. Because they make different mistakes, pooling their answers cancels out the random errors and keeps the shared signal. That is the whole idea — everything below is just three ways to assemble and run the panel.

• • Bagging → many copies of one model on random data slices, averaged (Random Forest)
• • Boosting → models in sequence, each fixing the last one's errors (AdaBoost, XGBoost)
• • Voting → diverse model types cast a majority vote
• • Stacking → a meta-model learns how to best combine the others

1 Why Ensembles Beat Single Models

A single model has a fixed weakness . A deep decision tree memorises noise (high variance — it swings wildly with the data). A shallow one is too simple to capture the pattern (high bias — it underfits). You can't escape both at once with one model.

Ensembles sidestep this. If three models each get a question right 70% of the time and their mistakes are independent , a majority vote is right far more than 70% of the time — because for the vote to be wrong, two models must fail on the same sample at once, which is much rarer. The key word is diverse : models that make the same mistakes give you nothing.

Run the worked example below. Three flawed models, none of them perfect, combine into a perfect ensemble on this set — purely because they err on different samples. This uses plain Python , no libraries.

2 Bagging & Random Forests — Cutting Variance

Bagging (short for Bootstrap Aggregating ) trains many copies of the same model type, each on a different random sample of the training data drawn with replacement (a "bootstrap" sample, where some rows appear twice and others not at all). You then average their predictions (or vote, for classification).

Because each model sees slightly different data, their over-fitting quirks point in different directions and cancel out when averaged. Bagging attacks the variance half of the bias–variance tradeoff — it makes an unstable, overfit-prone model far more stable.

A Random Forest is bagging applied to decision trees, with one extra twist: at each split, each tree may only consider a random subset of features . That forces the trees to be different from one another, boosting diversity even more. It's the go-to low-effort, high-accuracy baseline for tabular data.

Here is the same idea using scikit-learn's RandomForestClassifier inside a soft-voting ensemble:

3 Boosting — Cutting Bias, One Correction at a Time

Boosting flips bagging on its head. Instead of training models independently and in parallel, it trains them sequentially . Each new model focuses on the examples the previous models got wrong , so the team steadily chips away at the errors. Boosting attacks the bias half of the tradeoff: it turns a pile of "weak learners" (models barely better than guessing) into one strong learner.

• AdaBoost — re-weights the data after each round so misclassified points matter more; the next model concentrates on them.
• Gradient Boosting — each new tree is fit to the residuals (the leftover errors) of the current ensemble, nudging predictions toward the truth.
• XGBoost / LightGBM — highly optimised gradient-boosting libraries. They add regularisation, handle missing values natively, and train fast. XGBoost is the long-time Kaggle champion; LightGBM is faster on very large datasets thanks to histogram-based splits.

The price of boosting's accuracy: it can overfit if you use too many rounds or too high a learning rate, and it's sequential , so it's harder to parallelise than bagging. Here's gradient boosting in scikit-learn:

4 Voting & Stacking — Combining Different Model Types

Bagging and boosting build an ensemble from one kind of model. Voting and stacking instead combine different model types — say a logistic regression, a random forest, and an SVM — to maximise diversity.

• Hard voting — each model predicts a class; the majority wins (exactly the worked example above).
• Soft voting — average the predicted probabilities and pick the highest. Usually beats hard voting because confident models get more say.
• Stacking — train a small "meta-model" that takes the base models' predictions as input and learns the best way to blend them, instead of using a fixed rule like averaging.

Stacking is the most powerful but the most prone to leakage : the meta-model must be trained on held-out predictions (via cross-validation), never on predictions the base models made for data they were trained on — see the Common Errors below.

🎯 Your Turn #1: Weighted Average Ensemble

A plain average treats every model equally. But a stronger model should count for more. Fill in the three blanks to compute a weighted average , then run it and check against the expected output in the comments.

🎯 Your Turn #2: Weighted Majority Vote

Now do the classification version. Each model votes 0 or 1, but a more accurate model's vote carries more weight. Fill in the blanks to tally the weighted votes and pick the winner.

🧗 Mini-Challenge: A 5-Model Ensemble (support faded)

No blanks this time — just a comment outline. Build the whole majority-vote loop yourself for five models, then print the ensemble accuracy. The expected result is in the comments so you can self-check.

5 Common Errors (And How to Fix Them)

You ensemble five models and accuracy barely moves. They're all the same algorithm on the same data, so they make the same mistakes — averaging identical errors changes nothing.

✅ Fix: maximise diversity. Use different algorithms, different feature subsets (Random Forest does this), or different data samples (bagging). Diversity is the fuel of every ensemble.

Your gradient-boosting model scores 100% on training but slumps on the test set. Too many rounds or too high a learning_rate let it memorise noise.

✅ Fix: lower the learning_rate , cap n_estimators , use early_stopping on a validation set, and keep trees shallow ( max_depth=3 ). More small trees beat fewer big ones.

Your stacked model looks amazing in testing but flops in production. The meta-model was trained on predictions the base models made for their own training rows — it peeked at the answers.

✅ Fix: generate the meta-features with cross-validation (out-of-fold predictions only). Scikit-learn's StackingClassifier does this for you — don't hand-roll it unless you replicate the CV.

A 500-tree forest or a giant stack is accurate but answers each request too slowly for real time.

✅ Fix: prune the ensemble (fewer estimators), pick a faster library (LightGBM over a deep forest), or distil the ensemble into one small model. Accuracy that misses the latency budget is worthless in production.

📋 Quick Reference: Bagging vs Boosting

Aspect

Bagging

Boosting

How models train

In parallel, independently

Sequentially, each fixes the last

Data per model

Random bootstrap sample

Re-weighted toward past errors

Mainly reduces

Variance (overfitting)

Bias (underfitting)

Overfit risk

Low — very robust

Higher — needs tuning

Combine by

Averaging / majority vote

Weighted sum of learners

Top algorithms

Random Forest

AdaBoost, XGBoost, LightGBM

💡 Pro Tip: For tabular data, start with Random Forest as a no-fuss baseline, then try XGBoost or LightGBM when you need the last few points of accuracy. Deep learning usually only wins on images, text, and audio — on structured data, gradient boosting is still king.

❓ Frequently Asked Questions

🎉 Lesson Complete!

You can now explain bagging (parallel, variance-cutting — Random Forest), boosting (sequential, bias-cutting — AdaBoost, Gradient Boosting, XGBoost/LightGBM), and voting & stacking for mixing model types. You built a working majority vote and an averaging ensemble in plain Python, and you know the bias–variance reason ensembles beat single models.

🚀 Up next: Feature Engineering — the art of crafting powerful input features that make every one of these models sharper.

Practice quiz

Continue this course

• Previous: Dimensionality Reduction
• Next: Feature Engineering

Related lessons

• Decision Trees