Cross-Validation & ROC

Reviewed & published by Brayan K

Cross-validation and ROC analysis answer the question that matters most in modelling: how well will this model do on data it has never seen?

Part of the free R course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

By the end of this lesson you'll split data and run k-fold CV and LOOCV, spot overfitting, and read ROC curves, AUC, sensitivity, and specificity to judge a classifier honestly.

What You'll Learn in This Lesson

• Why train/test splits and CV prevent optimistic scores
• How k-fold cross-validation and LOOCV work
• What overfitting looks like in practice
• How to read the ROC curve and AUC
• What sensitivity and specificity measure
• How to compute ROC with pROC and CV with caret

1️⃣ Train/Test Split and Overfitting

Hold out part of the data, train on the rest, and compare errors. When the training error is far lower than the test error, the model is overfitting — memorizing noise instead of signal.

# The simplest guard against overfitting: a single train/test split.
set.seed(1)
data(mtcars)

n   <- nrow(mtcars)
idx <- sample(seq_len(n), size = 0.7 * n)   # 70% to train
train <- mtcars[idx, ]
test  <- mtcars[-idx, ]

fit  <- lm(mpg ~ wt + hp, data = train)

# Compare error on data the model saw vs data it did not:
train_rmse <- sqrt(mean((train$mpg - predict(fit, train))^2))
test_rmse  <- sqrt(mean((test$mpg  - predict(fit, test))^2))
c(train = round(train_rmse, 2), test = round(test_rmse, 2))

2️⃣ k-Fold Cross-Validation

A single split wastes data and varies with luck. trainControl(method = "cv", number = 10) rotates the validation role through 10 folds and averages the result. method = "LOOCV" takes it to the extreme.

library(caret)

# k-fold CV reuses ALL the data: every row is validated exactly once.
# Set method = "cv", number = 10 for 10-fold cross-validation.
ctrl <- trainControl(method = "cv", number = 10)

set.seed(1)
cv_fit <- train(mpg ~ wt + hp, data = mtcars,
                method = "lm", trControl = ctrl)

cv_fit$results[, c("RMSE", "Rsquared")]
# A single, more stable performance estimate than one split.

# LOOCV is k-fold with k = number of rows:
#   trainControl(method = "LOOCV")

3️⃣ The ROC Curve and AUC

For classifiers, pROC::roc() sweeps the decision threshold and plots sensitivity against 1 - specificity. The auc() summarizes the whole curve: 1.0 is perfect, 0.5 is chance.

library(pROC)

# A classifier outputs probabilities; the ROC curve shows the trade-off
# between catching positives (sensitivity) and false alarms (1 - specificity)
# across every possible threshold.
set.seed(1)
truth <- c(rep(1, 20), rep(0, 20))           # 20 positives, 20 negatives
score <- c(rnorm(20, 1.5), rnorm(20, 0))      # higher scores for positives

roc_obj <- roc(truth, score)
auc(roc_obj)                                  # area under the curve
# plot(roc_obj) draws the curve

4️⃣ Sensitivity & Specificity

At a fixed threshold, the confusion matrix gives Sensitivity (the true positive rate) and Specificity (the true negative rate). confusionMatrix() computes both.

library(caret)

# Confusion-matrix metrics summarize a classifier at one threshold.
#   Sensitivity = TP / (TP + FN)   -- found positives
#   Specificity = TN / (TN + FP)   -- correct negatives
pred  <- factor(c("yes","yes","no","no","yes","no"))
truth <- factor(c("yes","no","no","no","yes","yes"))

confusionMatrix(pred, truth, positive = "yes")
#   Sensitivity : 0.6667
#   Specificity : 0.6667

Your turn. Fill in the ___ blank, run it, and compare with the expected output.

library(caret)

# 🎯 YOUR TURN — replace each ___ then run it.

data(mtcars)

# 1) Ask caret for 5-fold cross-validation
ctrl <- trainControl(method = "cv", number = ___)   # 👉 replace ___ with 5

# 2) Cross-validate a linear model of mpg on wt + hp
set.seed(1)
fit <- train(mpg ~ wt + hp, data = mtcars, method = "lm", trControl = ctrl)

# 3) Print the cross-validated RMSE
cat("CV RMSE:", round(fit$results$RMSE, 3), "\n")

# ✅ Expected output (approximately):
#    CV RMSE: 2.64

Mini-Challenge: ROC and AUC

Build labels and scores, then compute the AUC with pROC and check it beats chance. AUC above 0.5 means the model ranks positives above negatives more often than not.

# 🎯 MINI-CHALLENGE: ROC and AUC for a classifier.
#
# 1. Build truth labels: 15 positives (1) and 15 negatives (0).
# 2. Build scores: rnorm(15, 1) for positives, rnorm(15, 0) for negatives.
# 3. With pROC, make an roc() object and print its AUC.
# 4. State whether AUC > 0.5 (better than chance).
#
# ✅ Example output:
#    AUC: 0.76
#    Better than chance: TRUE

# your code here

Common Errors (and the fix)

• Reporting training accuracy as the real score — always evaluate on a held-out fold or test set, never the data used to fit.
• Leaking the test set into preprocessing — scale and impute using only training data, or do it inside train().
• Forgetting to set a seed — folds and random scores change run to run without set.seed().
• Confusing sensitivity and specificity — sensitivity is about positives found; specificity is about negatives correctly rejected.

Pro Tips

• 💡 Default to 10-fold: it balances bias and variance for most datasets.
• 💡 Use AUC for imbalanced data: accuracy can look high while the model ignores the rare class.
• 💡 Pick a threshold deliberately: the ROC curve helps you trade sensitivity for specificity to fit the cost of each error.

📋 Quick Reference — CV & ROC

🎉 Lesson Complete!

• ✅ A held-out set gives an honest, non-optimistic score
• ✅ k-fold CV and LOOCV reuse all the data for stable estimates
• ✅ Overfitting shows as low train error but high test error
• ✅ The ROC curve and AUC summarize a classifier across thresholds
• ✅ Sensitivity and specificity describe it at one threshold

Practice quiz

Continue this course

• Previous: Bayesian Statistics in R
• Next: Interactive Plots with plotly — Add hover, zoom and interactivity with plot_ly() and ggplotly()
• Quick reference: R cheat sheet

Frequently asked questions

Why is cross-validation better than a single train/test split?

A single split gives one performance number that depends heavily on which rows happened to land in the test set — unlucky splits can mislead you. k-fold cross-validation rotates the validation role through every fold, so each observation is used for validation exactly once and for training the other times. Averaging the k scores gives a more stable, lower-variance estimate of how the model generalizes, and it uses all the data efficiently, which matters when datasets are small.

What is the difference between k-fold CV and LOOCV?

They are the same idea at different settings of k. Standard k-fold CV uses a modest k like 5 or 10: the data is split into k folds, and the model is trained k times. Leave-one-out cross-validation (LOOCV) sets k equal to the number of observations, so each fold is a single row — the model trains n times on n-1 rows. LOOCV has very low bias but high variance and can be expensive, while 5- or 10-fold CV usually offers a better bias-variance trade-off and is the common default.

What do the ROC curve and AUC actually tell me?

A classifier that outputs probabilities needs a threshold to turn scores into yes/no decisions. The ROC curve sweeps that threshold from strict to lenient and plots the true positive rate (sensitivity) against the false positive rate (1 - specificity) at each setting, showing the trade-off between catching positives and raising false alarms. The area under that curve, the AUC, summarizes the whole curve in one number: it is the probability that the model ranks a random positive above a random negative. AUC of 1.0 is perfect, 0.5 is chance.

How do sensitivity, specificity, and overfitting connect to all this?

Sensitivity (the true positive rate) and specificity (the true negative rate) describe a classifier at one chosen threshold, and they trade off against each other as you move the threshold along the ROC curve. Overfitting is the underlying danger these tools guard against: a model that memorizes training noise will show excellent sensitivity and specificity on its training data but poor numbers on a held-out fold. Cross-validation reveals that gap, and the ROC/AUC and confusion-matrix metrics quantify how well the model truly discriminates on data it has not seen.