Benchmarking with Criterion
Guessing at performance is how regressions sneak in. Criterion gives Rust statistically rigorous benchmarks — on stable — so you measure real timings, catch slowdowns, and prove your optimizations actually worked.
Learn Benchmarking with Criterion in our free Rust course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a quick…
Part of the free Rust course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
In this lesson you'll set up a benches/ target, write a Criterion benchmark with criterion_group! and criterion_main!, use b.iter and black_box correctly, and read the statistical reports.
What You'll Learn in This Lesson
1️⃣ Setting Up a benches/ Target
Criterion is a dev-dependency , and benchmark files live in a benches/ directory. A [[bench]] section with harness = false lets Criterion supply its own main .
With that in place, cargo bench compiles in release mode and runs every benchmark target.
2️⃣ Writing a Benchmark
A benchmark function takes &mut Criterion , registers work with bench_function , and times the closure passed to b.iter . The criterion_group! and criterion_main! macros build the harness.
Criterion reports a low/median/high interval, not a single number, because it samples many runs and analyzes the distribution statistically.
3️⃣ black_box and Avoiding Pitfalls
The biggest benchmarking trap is the optimizer deleting work whose result is unused, producing impossibly fast times. black_box is an opaque barrier: wrap the input so it can't be precomputed, and let the output escape so it can't be removed.
When the html_reports feature is on, Criterion also writes plots and a comparison against the previous run under target/criterion .
Mini-Challenge: Benchmark a Reversal
Write a small function and a Criterion benchmark that times it with black_box .
Common Errors (and the fix)
- Implausibly fast times (~0 ns) — The optimizer removed the work; wrap inputs/outputs in black_box .
- "multiple main functions" — Set harness = false on the [[bench]] target.
- "#[bench] is unstable" — The built-in benchmarker needs nightly; use Criterion on stable instead.
- Benchmark not found — Put the file in benches/ and match the name in Cargo.toml.
Pro Tips
- 💡 Always black_box: wrap inputs and let results escape so you measure real work.
- 💡 Trust the interval, not one number: Criterion's statistics beat a single stopwatch reading.
- 💡 Compare across runs: Criterion's saved baselines flag regressions before they ship.
📋 Quick Reference — Criterion
Goal
Rust Syntax
Dev dependency
[dev-dependencies] criterion = "0.5"
Bench target
[[bench]] name = "..." harness = false
Register a bench
c.bench_function("name", |b| { })
Time a routine
b.iter(|| work())
Stop optimizer
black_box(value)
Run
cargo bench
Frequently Asked Questions
🎉 Lesson Complete!
- ✅ Criterion brings statistically rigorous benchmarking to stable Rust
- ✅ Benchmarks live in benches/ , wired with criterion_group! / criterion_main!
- ✅ b.iter times the routine; black_box keeps the optimizer honest
- ✅ cargo bench produces intervals and HTML reports; the built-in #[bench] needs nightly
- ✅ Next lesson: Cargo workspaces and publishing
Practice quiz
What is Criterion used for in Rust?
- Statistics-driven benchmarking on stable Rust
- Unit testing
- Code formatting
- Async scheduling
Answer: Statistics-driven benchmarking on stable Rust. Criterion is a benchmarking library that runs on stable Rust and analyzes timings statistically.
Where do Criterion benchmark files conventionally live?
- In src/
- In a tests/ directory
- In a benches/ directory
- In examples/
Answer: In a benches/ directory. Cargo looks for benchmark targets in the benches/ directory at the crate root.
Which command runs your benchmarks?
- cargo test
- cargo bench
- cargo run --bench
- cargo check
Answer: cargo bench. cargo bench compiles and runs the benchmark targets, including Criterion ones.
What does black_box do in a benchmark?
- Logs results
- Hides output
- Stops the timer
- Prevents the optimizer from eliminating the code under test
Answer: Prevents the optimizer from eliminating the code under test. black_box is an opaque barrier that stops the compiler from optimizing away the work being measured.
What does Bencher::iter (the closure passed to b.iter) do?
- Repeatedly runs the routine so Criterion can time it
- Sorts the input
- Spawns threads
- Runs the routine once
Answer: Repeatedly runs the routine so Criterion can time it. b.iter runs the supplied closure many times so Criterion can collect a reliable timing sample.
What pair of macros wires up a Criterion benchmark binary?
- main! and group!
- criterion_group! and criterion_main!
- bench_start! and bench_end!
- setup! and teardown!
Answer: criterion_group! and criterion_main!. criterion_group! collects benchmark functions and criterion_main! generates the harness main.
Why is Criterion's built-in alternative, #[bench], not commonly used?
- It needs the internet
- It is deprecated and removed
- It only works on Windows
- It requires unstable nightly Rust
Answer: It requires unstable nightly Rust. The built-in #[bench] attribute and test::Bencher are unstable and only available on nightly.
What kind of report can Criterion generate after a run?
- A SQL dump
- A PDF invoice
- HTML reports with plots and statistics
- A Dockerfile
Answer: HTML reports with plots and statistics. Criterion produces detailed HTML reports including plots, distributions, and regression analysis.
How does Criterion help detect performance changes between runs?
- It deletes old results
- It compares against saved previous results and reports regressions/improvements
- It only prints the latest time
- It requires a paid service
Answer: It compares against saved previous results and reports regressions/improvements. Criterion stores baseline results and statistically compares new runs to flag regressions or speedups.
A common pitfall when benchmarking pure functions is...
- The compiler optimizing the work away because the result is unused
- Forgetting to import std
- Running on a laptop
- Using too many threads
Answer: The compiler optimizing the work away because the result is unused. If a result is never used, the optimizer can delete the computation; wrapping inputs/outputs in black_box prevents this.
Continue this course
- Previous: Procedural & Derive Macros
- Next: Cargo Workspaces & Publishing