Model Selection & Tuning

Stop guessing which model is best. By the end you'll split data three ways, run k-fold cross-validation by hand and with scikit-learn, read the bias-variance tradeoff, tune hyperparameters with grid, random and Bayesian search, and avoid the traps that make a model look better than it really is.

Learn Model Selection & Tuning in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

🌍 Real-World Analogy

Think of hiring for a job by running fair trials. You have several candidates (models and settings) and you want the one who'll actually perform — not the one who happened to ace a single easy question.

So you give every candidate the same set of fair, varied tasks (that's cross-validation — rotating which slice of data is the test). You compare them on those tasks (that's the search over hyperparameters). And crucially, you keep one final, unseen task locked away to confirm your pick really is good (that's the held-out test set). If you kept re-running the same trial and tweaking until someone passed it, you'd just be fitting your choice to that one quiz — which is exactly the overfitting trap this lesson teaches you to dodge.

1 Three Sets: Train, Validation, Test

A model that scores its own homework always claims an A. To get an honest measure, you split your data into three roles before you start:

• Training set (~60–80%) — the model learns its parameters from these.
• Validation set (~10–20%) — used to choose between models and settings while you tune.
• Test set (~10–20%) — locked away, touched once at the very end to report real performance.

The validation set is where you experiment; the test set is the final exam you only sit once. If you tune against the test set, its score stops being honest — you've turned your final exam into practice. Keep it sealed until the end.

2 k-Fold Cross-Validation: One Score You Can Trust

A single train/validation split gives you one number, and that number depends on which rows happened to land in the validation set — you could get lucky or unlucky. k-fold cross-validation fixes this. You chop the data into k equal chunks ("folds"), then run k rounds: each round, one fold is the test set and the other k − 1 folds are the training set.

• Every row is used for testing exactly once and for training k − 1 times.
• You average the k scores for a reliable estimate, and look at the spread (std) to judge stability.
• k = 5 or k = 10 are the usual choices.

The worked example below splits a list into folds by hand, runs the rotation loop, averages the scores, and then picks the best of a few candidate scores — exactly the model-selection move.

🎯 Your Turn: split a list into k folds

3 The Real Way: cross_val_score

You now understand the loop under the hood, so in real projects you let scikit-learn run it. cross_val_score(model, X, y, cv=5) does the whole rotation for you and returns one score per fold. Report the mean as your estimate and the standard deviation as the spread.

Read the example below as a worked walkthrough — it needs scikit-learn installed locally to run, and the comments show the expected output.

4 The Bias-Variance Tradeoff

Every model error splits into two parts, and they pull in opposite directions:

• Bias — error from a model that's too simple . It underfits : misses real patterns, like a straight line through clearly curved data. High train error and high test error.
• Variance — error from a model that's too complex . It overfits : memorises noise, so it swings wildly on new data. Low train error but high test error.

You can't zero out both: adding flexibility lowers bias but raises variance, and vice versa. The goal is the sweet spot in the middle. Here's how to diagnose where you are:

• High train error + high test error → underfitting (add complexity/features)

• Low train error + high test error → overfitting (regularise/simplify/more data)

• Low train error + low test error → good fit 🎯

5 Hyperparameter Tuning: Grid, Random, Bayesian

A hyperparameter is a setting you pick before training — tree depth, learning rate, number of neighbours. The model doesn't learn these; you do, by trying values and cross-validating each. Three strategies:

• Grid search — try every combination on a grid. Exhaustive and reproducible, but combinations grow multiplicatively, so it's slow.
• Random search — sample N random combinations. Usually finds a near-best setting far faster, because only a few hyperparameters really matter.
• Bayesian optimisation — use results from past trials to choose the next combination intelligently (tools like Optuna). Fewest trials of all.

Each combination is scored with cross-validation, and the best one wins. The read-along below runs GridSearchCV for real.

🎯 Your Turn: pick the best candidate

6 Don't Overfit to the Validation Set

Here's a subtle trap. Every time you tweak a model and check the validation score, you're letting that set influence your choices. Do it hundreds of times and you start overfitting to the validation set — your "best" model is now tuned to the quirks of that particular slice, and its score is optimistic.

• Keep a truly held-out test set you evaluate only once, at the end.
• Limit how many configurations you try; more trials means more chance to fit noise.
• Prefer cross-validation over a single validation split — it's harder to game.

Common Errors (And How to Fix Them)

You peek at the test score, tweak the model to improve it, and repeat. The test set is now part of training and its score is a fantasy.

✅ Fix: lock the test set away. Tune only on the validation set or with cross-validation, and touch the test set exactly once, at the very end.

You report a single train/test number. It looked great by luck, then the model flops in production.

✅ Fix: use k-fold cross-validation ( cross_val_score(model, X, y, cv=5) ) and report the mean, not one lucky split.

❌ Ignoring variance — reporting only the mean

Two models both average 0.85, so you call it a tie — but one ranges 0.84–0.86 and the other swings 0.70–0.99.

✅ Fix: always report the standard deviation alongside the mean ( scores.std() ). Low spread means a result you can trust.

You scale or impute on the whole dataset before splitting, so every fold has secretly seen its own test data. Scores look amazing and are wrong.

✅ Fix: do all preprocessing inside the CV loop. Wrap it in a Pipeline and pass the pipeline to cross_val_score / GridSearchCV .

📋 Quick Reference

Technique

Code / Idea

Meaning

Train/val/test split

~70 / 15 / 15

Learn, tune, then score once

k-fold CV

cross_val_score(m, X, y, cv=5)

Every row tested once; average the folds

Bias (underfit)

high train + high test error

Too simple — add complexity

Variance (overfit)

low train + high test error

Too complex — regularise/simplify

Grid search

GridSearchCV(...)

Try every combo (thorough, slow)

Random search

RandomizedSearchCV(...)

Sample N combos (fast, near-best)

Avoid leakage

Pipeline(scaler, model)

Fit preprocessing inside each fold

Frequently Asked Questions

Mini-Challenge: is this CV result trustworthy?

No blanks this time — just a brief and a comment outline. Take five per-fold scores, compute the mean and the spread, and warn when the variance is high. Check your answer against the expected result in the comments.

🎉 Lesson Complete!

You can now select and tune models the right way: you split data into train, validation, and test; you run k-fold cross-validation by hand and with cross_val_score ; you read the bias-variance tradeoff to diagnose under- and overfitting; you tune hyperparameters with grid, random, and Bayesian search; and you sidestep the traps of tuning on the test set and data leakage.

🚀 Up next: Time Series Forecasting — predicting the future from sequential data, where the train/test split has to respect the order of time.

Practice quiz

Continue this course

• Previous: Feature Engineering
• Next: Time Series Forecasting