Pairwise comparisons, such as human votes between two responses, can be turned into a ranking. Bradley–Terry and Elo are the two standard approaches.
The Bradley–Terry model
Give each model a strength parameter. The probability that model A beats model B depends on the difference in their log-strengths through a logistic function. Fit the parameters by maximum likelihood on all observed comparisons.
Elo
Elo is an online version. After each match, the winner's rating increases and the loser's decreases, in proportion to how surprising the result was. It is simple and works for streaming data, but results depend on the order of matches and the update step size.
Why Bradley–Terry is often preferable for fixed data
If you have a complete batch of comparisons and the models do not change, fitting Bradley–Terry directly is more stable and order-independent. It also yields standard errors and confidence intervals.
Rank uncertainty
Ratings are estimates. Two models with close ratings may be impossible to rank reliably. Bootstrapping the comparisons gives a distribution over ranks and shows which positions are actually settled.
Pitfalls
- Uneven matchups can bias results if some models face easier opponents.
- Ties need explicit handling.
- Style preferences, such as longer or more polished answers, can influence votes. Control for them where possible.
What to report
Show ratings with intervals, the number of comparisons per model, and any steps taken to adjust for known biases.
Continue with the book
Explore the full treatment, worked examples, and exercises in Statistical Foundations for LLM Development and Evaluation.
Adapted from Statistical Foundations for LLM Development and Evaluation, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.