Gradient Boosting (XGBoost & LightGBM)

Chain many weak trees into one of the most accurate models for tabular data. Learn how each tree fixes the last one's errors, and how learning_rate and n_estimators control the fit.

Learn Gradient Boosting (XGBoost & LightGBM) in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🎯 Real-World Analogy: A Team of Editors

Imagine a draft passed down a line of editors. The first writes a rough version. The second doesn't rewrite from scratch — they only fix the mistakes the first one left. The third fixes whatever errors remain after that. Each editor makes a small correction to the previous result.

That is gradient boosting : each new weak tree is trained on the errors (residuals) still left by the ensemble so far, and a small learning rate keeps every correction modest so the team doesn't over-correct.

1 Boosting vs Bagging

A random forest uses bagging : it trains many independent trees in parallel on random samples and averages them. The trees never coordinate; averaging cancels their random errors and reduces variance.

Boosting is the opposite philosophy. Trees are added one at a time , and each new tree is trained specifically to fix the errors the current ensemble still makes. This gradually reduces bias and produces a very strong final model.

• Start with a simple prediction (often the mean).
• Compute the residuals — what the model still gets wrong.
• Train a small weak tree to predict those residuals.
• Add a fraction (the learning rate) of that tree to the model.
• Repeat for n_estimators rounds.

2 Watching Residuals Shrink

Let's see boosting in action. We start by predicting the mean for everyone, then each round nudges the prediction toward the true values by a fraction of the residual. Watch the predictions march toward the targets.

3 learning_rate & n_estimators

Two hyperparameters dominate boosting. n_estimators is how many trees (rounds) you add. learning_rate (shrinkage) scales how much each tree contributes — a smaller rate means smaller, safer steps.

They interact: a smaller learning rate needs more trees to reach the same fit. The classic recipe is a small learning rate (like 0.05), a generous n_estimators, and early stopping to halt once a validation set stops improving.

🌍 Worked Example: The Real Tool (XGBoost / LightGBM)

In practice you call a library. Here's XGBoost's XGBClassifier ; LightGBM's LGBMClassifier is nearly identical. Study it (it isn't runnable in the in-browser sandbox) and notice the small learning_rate , larger n_estimators , shallow max_depth , and subsample for regularisation.

LightGBM trains faster on large data thanks to histogram-based splits and leaf-wise growth, while XGBoost is the well-tested competition standard.

🎯 Your Turn 1: One Boosting Round

Fill in the blanks to compute the residual and take one boosting step. The expected output is in the comments.

🎯 Your Turn 2: Pick Safer Settings

Decide whether a (learning_rate, n_estimators) combination is the safer choice. Fill in the two blanks to match the rule of thumb.

🎯 Mini-Challenge: Boost Toward 100

Run three boosting rounds toward a target value and print the prediction each round. Only a comment outline is provided.

4 Common Errors (And How to Fix Them)

These are the classic boosting pitfalls. Watch for them.

Big steps plus many rounds rush past the signal and start fitting noise.

Without held-out data you can't tell when boosting starts overfitting.

Treating boosting like a random forest (deep independent trees) loses its whole advantage.

✅ Fix: keep the base trees shallow (weak learners):

📋 Quick Reference

Concept

What It Is

Key Point

Boosting

Sequential error-correcting trees

Reduces bias

Bagging

Parallel independent trees

Reduces variance

Weak learner

Shallow base tree

Corrected by later trees

Residual

Leftover error each round

Next tree fits it

learning_rate

Shrinkage per tree

Smaller = needs more trees

n_estimators

Number of boosting rounds

Too many overfits

Early stopping

Halt on validation plateau

Prevents overfitting

XGBoost / LightGBM

Fast boosting libraries

Dominant on tabular data

❓ Frequently Asked Questions

🎉 Lesson Complete!

You now understand how boosting adds trees sequentially to fix residuals , how it differs from bagging , how learning_rate and n_estimators interact, and why XGBoost and LightGBM dominate tabular data.

🚀 Up next: Feature Engineering & Selection — the craft of giving your models better inputs.

Practice quiz

Continue this course

• Previous: Dimensionality Reduction with PCA
• Next: Feature Engineering & Selection