ANOVA & Model Comparison

Reviewed & published by Brayan K

ANOVA (Analysis of Variance) is a statistical test that uses an F-statistic to decide whether the means of three or more groups differ by more than random chance.

Part of the free R course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

By the end of this lesson you'll fit one-way and two-way ANOVAs with aov(), read every column of the ANOVA table, run TukeyHSD to find which groups differ, and use anova() to compare nested models.

What You'll Learn in This Lesson

• How aov() fits a one-way ANOVA
• What the F-statistic and Pr(>F) mean
• How to read every column of the ANOVA table
• How TukeyHSD() finds which groups differ
• How two-way ANOVA adds interactions
• How anova() compares nested lm models

1️⃣ One-Way ANOVA and the F-Test

One-way ANOVA tests whether a numeric outcome differs across the levels of a single grouping factor. You fit it with aov(outcome ~ group) and read the result with summary(). The key output is the F value and its Pr(>F) p-value.

# One-way ANOVA: does a numeric outcome differ across GROUPS?
# Here: does plant growth differ by fertilizer type?

growth <- c(4.2, 4.8, 5.1, 6.0, 6.4, 6.1, 5.0, 4.9, 5.3)
fert   <- factor(c("A","A","A","B","B","B","C","C","C"))

# aov() fits the model: outcome ~ grouping factor
fit <- aov(growth ~ fert)

# summary() prints the ANOVA table with the F-statistic and p-value
summary(fit)

# Read the table (top to bottom):
#   Df       = degrees of freedom (groups - 1, then residual)
#   Sum Sq   = variation explained by the factor vs. left over
#   Mean Sq  = Sum Sq / Df
#   F value  = between-group MS / within-group MS
#   Pr(>F)   = p-value. Small (< 0.05) => groups really differ.

2️⃣ Which Groups Differ? TukeyHSD

A significant ANOVA only says "some group differs." To find which pairs differ, run TukeyHSD() on the fitted model. It reports each pairwise difference with a confidence interval and an adjusted p-value, correcting for multiple comparisons.

# A significant ANOVA says "at least one group differs" — but WHICH ones?
# TukeyHSD() compares every pair while controlling the error rate.

growth <- c(4.2, 4.8, 5.1, 6.0, 6.4, 6.1, 5.0, 4.9, 5.3)
fert   <- factor(c("A","A","A","B","B","B","C","C","C"))
fit    <- aov(growth ~ fert)

TukeyHSD(fit)

# The output lists each pair (B-A, C-A, C-B) with:
#   diff  = estimated difference in means
#   lwr/upr = confidence interval for that difference
#   p adj = adjusted p-value. If the interval excludes 0
#           (and p adj < 0.05), that pair differs significantly.

3️⃣ Two-Way ANOVA and Comparing Models

Two-way ANOVA studies two factors at once and whether they interact. The formula y ~ a * b expands to both main effects plus the a:b interaction. A significant interaction means the effect of one factor depends on the other.

# Two-way ANOVA: TWO factors at once, plus their interaction.
# Does yield depend on fertilizer AND watering — and do they interact?

# yield for two fertilizers (A/B) under two watering levels (low/high):
df <- data.frame(
  yield = c(5.0, 5.1, 5.2, 5.3, 5.0, 8.0, 5.4, 8.4),   # water only helps fertilizer B
  fert  = factor(rep(c("A", "B"), each = 4)),
  water = factor(rep(c("low", "high"), times = 4))
)
# fert + water        tests each factor's MAIN effect
# fert:water          tests whether the effect of one DEPENDS on the other

fit2 <- aov(yield ~ fert + water + fert:water, data = df)
summary(fit2)

# Shorthand: fert * water expands to fert + water + fert:water
fit2 <- aov(yield ~ fert * water, data = df)
summary(fit2)

# In the table, a small Pr(>F) on the 'fert:water' row means the
# interaction matters — you can't talk about fert without mentioning water.

Separately, anova() compares two nested linear models — a simpler one against a richer one — to decide whether the extra predictors earn their keep.

# anova() COMPARES two nested linear models with an F-test.
# "Nested" = the simpler model is a special case of the bigger one.

simple  <- lm(mpg ~ wt, data = mtcars)            # weight only
bigger  <- lm(mpg ~ wt + hp, data = mtcars)       # weight + horsepower

# Does adding horsepower significantly improve the fit?
anova(simple, bigger)

# The table's bottom row gives an F-statistic and Pr(>F).
# Small p => the extra term (hp) earns its place; keep the bigger model.
# Large p => the simpler model is just as good; prefer it (Occam's razor).

Your turn. Fill in the ___ blank, run it, and interpret the table.

# 🎯 YOUR TURN — replace each ___ then run it.

# Three teaching methods, exam scores for 3 students each
score  <- c(70, 75, 72, 85, 88, 84, 78, 80, 79)
method <- factor(c("X","X","X","Y","Y","Y","Z","Z","Z"))

# 1) Fit a one-way ANOVA of score by method
fit <- ___(score ~ method)   # 👉 replace ___ with aov

# 2) Print the ANOVA table
summary(fit)

# 3) If it's significant, see which pairs differ
TukeyHSD(fit)

# 👉 In the summary table, look at Pr(>F). Is it below 0.05?
#    If so, method matters. Then read TukeyHSD to see WHICH methods differ.

Mini-Challenge: Is the Bigger Model Worth It?

Fit two nested models on mtcars and let anova() referee. The lesson is the discipline of adding complexity only when the data justify it.

# 🎯 MINI-CHALLENGE: compare two models on built-in data.
#
# Using the built-in 'mtcars' dataset:
#   1. Fit  m1 <- lm(mpg ~ wt, data = mtcars)
#   2. Fit  m2 <- lm(mpg ~ wt + cyl + hp, data = mtcars)
#   3. Run  anova(m1, m2)  to test whether the extra terms help.
#   4. In one sentence, state your conclusion from Pr(>F):
#      - if p < 0.05, the richer model fits significantly better
#      - if p >= 0.05, prefer the simpler m1
#
# Bonus: run summary(m2) and note which individual predictors
#        have significant t-tests (their own Pr(>|t|) < 0.05).

# your code here

Common Errors (and the fix)

• Grouping variable isn't a factor — ❌ numeric codes get treated as continuous. ✅ Wrap it in factor().
• Reading aov() without summary() — ❌ printing the bare fit hides the F-test. ✅ Call summary(fit) for the table.
• Skipping the post-hoc test — ❌ claiming a specific pair differs from a significant omnibus alone. ✅ Confirm with TukeyHSD().
• Comparing non-nested models with anova() — ❌ the F-test is invalid. ✅ Compare AIC() or BIC() instead.

Pro Tips

• 💡 Check assumptions: plot(fit) shows residual diagnostics; ANOVA assumes roughly normal, equal-variance residuals.
• 💡 ANOVA is regression: aov(y ~ g) is just lm(y ~ g) viewed through the variance lens.
• 💡 Unequal variance? use oneway.test() (Welch's ANOVA), which doesn't assume equal spread.

📋 Quick Reference — ANOVA

🎉 Lesson Complete!

• ✅ aov(y ~ group) fits a one-way ANOVA
• ✅ The F value and Pr(>F) test whether group means differ
• ✅ TukeyHSD() reveals which specific pairs differ
• ✅ a * b adds an interaction in two-way ANOVA
• ✅ anova(m1, m2) compares nested models with an F-test

Practice quiz

Continue this course

• Previous: Text Wrangling with stringr
• Next: Time Series (ts & forecasting basics) — Build ts objects, decompose trend and seasonality, and forecast
• Quick reference: R cheat sheet

Frequently asked questions

What does the F-statistic in an ANOVA actually measure?

The F-statistic is a ratio of two kinds of variation: the variation between group means divided by the variation within groups. If the groups truly differ, the between-group part is large relative to the noise within groups, pushing F well above 1. The p-value (Pr(>F)) then tells you how likely an F that large would be if all group means were actually equal. A small p-value (typically < 0.05) is evidence that at least one group differs.

ANOVA told me groups differ — why do I still need TukeyHSD?

A significant ANOVA is an omnibus result: it says 'somewhere among these groups there is a real difference' but not where. TukeyHSD (Tukey's Honest Significant Differences) then compares every pair of groups while adjusting for the fact that you're doing many comparisons at once. Without that adjustment, testing many pairs separately would inflate your false-positive rate. Tukey gives you per-pair differences, confidence intervals, and adjusted p-values so you can name the specific groups that differ.

What is the interaction term fert:water in a two-way ANOVA?

An interaction asks whether the effect of one factor depends on the level of another. fert:water tests whether the difference between fertilizers changes depending on the watering level. If that row is significant, you can't summarize each factor on its own — the story is 'fertilizer B helps, but only when watering is high,' for example. The shorthand fert * water automatically includes both main effects and the interaction.

How is anova() different from aov()?

aov() fits an ANOVA model and you typically call summary() on it to read the table. anova(), by contrast, is most useful for comparing two (or more) nested models — for example two lm() fits where one adds extra predictors. It performs an F-test asking whether the extra terms significantly improve the fit. So aov() answers 'do these groups differ?' while anova(model1, model2) answers 'is the bigger model worth the added complexity?'