If you run enough comparisons, some will look significant by chance. Leaderboards and ablation studies make this easy to do by accident.
The problem
At a 5% significance level, one test in twenty produces a false positive when there is no real effect. Compare 20 model variants and you should expect about one spurious win. Selecting the best of many and reporting it as if it were the only candidate inflates results.
Controlling errors
- Bonferroni divides the significance level by the number of tests. It is simple and conservative.
- Holm is a stepwise method that is uniformly more powerful than Bonferroni.
- Benjamini–Hochberg controls the false discovery rate, the expected share of false findings among those declared significant.
Winner's curse
The top-ranked model on a leaderboard is partly lucky. When re-evaluated on fresh data, it tends to score lower than before. The more candidates, the larger the effect.
Effect sizes matter
A tiny difference can be statistically significant with a huge sample but irrelevant in practice. Report effect sizes and consider whether they matter.
Practical habits
- Pre-register the comparisons you plan to make.
- Hold out a final test set that is used once.
- Count every variant you tried, including discarded ones.
- Report the ranking uncertainty, not just the order.
Bottom line
A leaderboard position is a hypothesis, not a verdict, until it survives fresh data and an honest accounting of how many options were tried.
Continue with the book
Explore the full treatment, worked examples, and exercises in Statistical Foundations for LLM Development and Evaluation.
Adapted from Statistical Foundations for LLM Development and Evaluation, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.