A score with no uncertainty attached invites over-reading. Confidence intervals are the simplest way to show how much a benchmark number can be trusted.
The simple binomial case
For accuracy on n independent questions, the standard error is roughly √(p(1 − p)/n). With 100 questions and an accuracy of 0.78, that is about 0.041, so a rough 95% interval is 0.78 ± 0.08, or about 70% to 86%.
Better intervals for extreme scores
The normal approximation behaves poorly near 0% or 100% and for small n. Wilson or Clopper–Pearson intervals handle those cases more reliably.
When questions are not independent
Benchmarks often contain clusters of related items, such as several questions about the same passage. Treating them as independent makes intervals too narrow. Use cluster-robust standard errors or resample at the cluster level.
The bootstrap
Resampling the evaluation items with replacement, recomputing the score, and taking percentiles gives an interval without a formula. It adapts to complex metrics such as F1 or pass@k.
Reporting tips
- State how the interval was computed.
- Do not read overlapping intervals as proof of no difference. For comparisons, use a paired analysis.
- Include repeated runs when generation is stochastic.
The result
Intervals turn a single number into an honest range, which makes it easier to see when a benchmark is too small to answer the question you are asking.
Continue with the book
Explore the full treatment, worked examples, and exercises in Statistical Foundations for LLM Development and Evaluation.
Adapted from Statistical Foundations for LLM Development and Evaluation, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.