When two models answer the same questions, their results are linked. A paired analysis uses that link to produce sharper conclusions.
Why pairing helps
Some questions are hard for every model, and others are easy. That variation is shared. Comparing each model's overall accuracy ignores the link and mixes difficulty differences into the noise. Looking at the difference on each question cancels most of it.
The simple contingency view
For binary correct/incorrect results, count four groups: both right, both wrong, only model A right, only model B right. Only the disagreements carry information about which model is better.
McNemar's test
McNemar's test asks whether the two disagreement counts are about equal. If A wins on 40 questions and B wins on 20, the test evaluates whether such an imbalance would be likely if the models were equally good.
Paired intervals for continuous scores
For graded or numeric scores, compute the difference on each item. Then form a confidence interval for the mean difference, or use a paired bootstrap or a permutation test.
Practical guidance
- Evaluate both models on identical prompts, settings, and graders.
- Keep item IDs so results can be aligned.
- Report the estimated difference and its interval, not only whether a test passed.
- Think about whether the difference is large enough to matter.
Result
Paired designs give more power for the same number of questions, which is often the cheapest improvement you can make to an evaluation.
Continue with the book
Explore the full treatment, worked examples, and exercises in Statistical Foundations for LLM Development and Evaluation.
Adapted from Statistical Foundations for LLM Development and Evaluation, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.