A common failure in evaluation is a test set too small to detect the difference that matters. Power analysis lets you plan before collecting data.
Two kinds of errors
A false positive declares a difference when none exists. A false negative misses a real difference. Power is the probability of detecting an effect of a given size, and it depends on the effect size, the noise, and the number of samples.
A rough calculation
For estimating one accuracy to within ± m, you need about n ≈ 1.96² × p(1 − p) / m². With p near 0.5 and a margin of 0.03, that is roughly 1,067 questions. Halving the margin requires four times as many.
Comparing two models
If you compare two models on the same questions, the relevant noise is the variability of the per-question differences, not of each score separately. Paired designs often need far fewer samples than unpaired ones, because question difficulty cancels out.
Plan around the smallest effect worth detecting
Decide in advance what improvement would change a decision, such as 1 percentage point or 3 percentage points. Then compute how many items you need to detect it reliably, commonly with 80% power.
When data is limited
- Combine related benchmarks carefully.
- Increase the number of items before increasing the number of repeated runs, since question variation is usually larger.
- Be explicit that small tests can only detect large differences.
Takeaway
Choose the sample size before the experiment, not after seeing a disappointing result.
Continue with the book
Explore the full treatment, worked examples, and exercises in Statistical Foundations for LLM Development and Evaluation.
Adapted from Statistical Foundations for LLM Development and Evaluation, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.