Benchmarking with Criterion

Reviewed & published by Brayan K

Guessing at performance is how regressions sneak in. Criterion gives Rust statistically rigorous benchmarks — on stable — so you measure real timings, catch slowdowns, and prove your optimizations actually worked.

Part of the free Rust course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

In this lesson you'll set up a benches/ target, write a Criterion benchmark with criterion_group! and criterion_main!, use b.iter and black_box correctly, and read the statistical reports.

What You'll Learn in This Lesson

• Adding criterion and a benches/ target
• criterion_group! and criterion_main!
• Timing routines with b.iter
• Stopping the optimizer with black_box
• Running cargo bench and reading reports
• Why the built-in #[bench] needs nightly

1️⃣ Setting Up a benches/ Target

Criterion is a dev-dependency, and benchmark files live in a benches/ directory. A [[bench]] section with harness = false lets Criterion supply its own main.

# Criterion benchmarks live in benches/ and are declared in Cargo.toml.
# Cargo.toml:

[dev-dependencies]
criterion = "0.5"

# Declare the benchmark target. harness = false lets Criterion provide main.
[[bench]]
name = "my_benchmark"
harness = false

# File layout:
#   my_crate/
#     src/lib.rs
#     benches/my_benchmark.rs   <- the benchmark file

With that in place, cargo bench compiles in release mode and runs every benchmark target.

2️⃣ Writing a Benchmark

A benchmark function takes &mut Criterion, registers work with bench_function, and times the closure passed to b.iter. The criterion_group! and criterion_main! macros build the harness.

// benches/my_benchmark.rs
// A complete Criterion benchmark for a fibonacci function.
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn fibonacci(n: u64) -> u64 {
    match n {
        0 => 0,
        1 => 1,
        _ => fibonacci(n - 1) + fibonacci(n - 2),
    }
}

fn bench_fib(c: &mut Criterion) {
    // b.iter runs the closure many times so Criterion can time it.
    // black_box hides the input so the optimizer can't precompute it.
    c.bench_function("fib 20", |b| {
        b.iter(|| fibonacci(black_box(20)))
    });
}

// These macros generate the benchmark harness and its main().
criterion_group!(benches, bench_fib);
criterion_main!(benches);

Criterion reports a low/median/high interval, not a single number, because it samples many runs and analyzes the distribution statistically.

3️⃣ black_box and Avoiding Pitfalls

The biggest benchmarking trap is the optimizer deleting work whose result is unused, producing impossibly fast times. black_box is an opaque barrier: wrap the input so it can't be precomputed, and let the output escape so it can't be removed.

// Why black_box matters: without it, the optimizer can delete your work.
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn sum_to(n: u64) -> u64 {
    (1..=n).sum()
}

fn bench_sum(c: &mut Criterion) {
    c.bench_function("sum_to 1000", |b| {
        // WRONG: a constant unused result may be optimized to nothing.
        //   b.iter(|| sum_to(1000));
        //
        // RIGHT: black_box the input AND let the result escape.
        b.iter(|| black_box(sum_to(black_box(1000))))
    });
}

criterion_group!(benches, bench_sum);
criterion_main!(benches);

When the html_reports feature is on, Criterion also writes plots and a comparison against the previous run under target/criterion.

Mini-Challenge: Benchmark a Reversal

Write a small function and a Criterion benchmark that times it with black_box.

// 🎯 MINI-CHALLENGE: benchmark a string reversal.
//
// 1. Write  fn reverse(s: &str) -> String  using chars().rev().collect().
// 2. In a bench fn, call c.bench_function("reverse", |b| ...).
// 3. Inside b.iter, call reverse(black_box("hello world")).
// 4. Wire it up with criterion_group! and criterion_main!.

use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn reverse(s: &str) -> String {
    // your code here
    String::new()
}

// add bench_reverse, criterion_group!, and criterion_main! yourself

Common Errors (and the fix)

• Implausibly fast times (~0 ns) — The optimizer removed the work; wrap inputs/outputs in black_box.
• "multiple main functions" — Set harness = false on the [[bench]] target.
• "#[bench] is unstable" — The built-in benchmarker needs nightly; use Criterion on stable instead.
• Benchmark not found — Put the file in benches/ and match the name in Cargo.toml.

Pro Tips

• 💡 Always black_box: wrap inputs and let results escape so you measure real work.
• 💡 Trust the interval, not one number: Criterion's statistics beat a single stopwatch reading.
• 💡 Compare across runs: Criterion's saved baselines flag regressions before they ship.

📋 Quick Reference — Criterion

🎉 Lesson Complete!

• ✅ Criterion brings statistically rigorous benchmarking to stable Rust
• ✅ Benchmarks live in benches/, wired with criterion_group!/criterion_main!
• ✅ b.iter times the routine; black_box keeps the optimizer honest
• ✅ cargo bench produces intervals and HTML reports; the built-in #[bench] needs nightly

Practice quiz

Continue this course

• Previous: Procedural & Derive Macros
• Next: Cargo Workspaces & Publishing — Multi-crate workspaces and publishing to crates.io with cargo publish
• Quick reference: Rust cheat sheet

Frequently asked questions

What is Criterion and why use it over the built-in benchmarker?

Criterion is the de-facto benchmarking library for Rust. Unlike a simple timer, it runs your code many times, collects a statistical sample of the measurements, estimates a confidence interval, and can compare a run against previously saved results to flag regressions or improvements. Crucially, it works on stable Rust. The standard library does include a built-in #[bench] attribute with test::Bencher, but that feature is unstable and only available on the nightly toolchain, so most projects cannot rely on it. Criterion fills that gap with a richer, statistically rigorous harness that any project on stable can use, which is why it has become the community standard.

How do I set up and run a Criterion benchmark?

Add criterion as a dev-dependency, then create a benches/ directory at your crate root and put a benchmark file there, for example benches/my_benchmark.rs. In Cargo.toml you declare the target with a [[bench]] section that sets name = "my_benchmark" and harness = false; the harness = false tells Cargo not to use the default test harness so Criterion can supply its own main via the criterion_main! macro. Inside the file you write one or more functions that take &mut Criterion, register them with criterion_group!, and finish with criterion_main!. Running cargo bench compiles in release mode and executes every benchmark, printing timing estimates to the terminal.

What do b.iter and black_box actually do?

The closure you pass to c.bench_function receives a Bencher, and calling b.iter(|| ...) runs the supplied routine repeatedly so Criterion can gather enough samples to time it accurately — you measure the inner closure, not the surrounding setup. black_box is an optimization barrier: it takes a value and returns it in a way the compiler must treat as opaque, so the optimizer cannot fold a constant input into a precomputed answer or, worse, delete the computation entirely because its result is never used. Wrapping the inputs (and often the output) of the code under test in black_box is what keeps a microbenchmark honest.

What output does Criterion produce and what pitfalls should I watch for?

On the terminal Criterion prints a time estimate as a low/median/high interval and notes outliers; when the 'html_reports' feature is enabled it also writes detailed HTML reports under target/criterion with plots of the timing distribution, line charts across runs, and a comparison against the previous baseline so you can see whether a change made things faster or slower. The biggest pitfall is letting the optimizer remove the work you meant to measure, producing implausibly tiny times — guard against it with black_box. Other pitfalls include benchmarking in debug mode (cargo bench uses release, but ad-hoc timing might not), measuring too little work per iteration so timer resolution dominates, and ignoring system noise; running on a quiet machine and trusting Criterion's statistics rather than a single number gives reliable results.