It is easier for people to say which of two responses is better than to assign each a number. The Bradley–Terry model converts those comparisons into a scalar reward.
The model
Give each response a score r. The probability that response A is preferred over response B is the sigmoid of the difference: σ(r(A) − r(B)). Only differences matter, so the scores are defined up to a shift.
The training loss
Given a dataset of chosen and rejected responses, maximize the likelihood of the observed preferences. That is the same as minimizing the negative log of σ(r(chosen) − r(rejected)). A neural network, often the language model with a scalar head, produces r.
What the model learns
The reward model compresses human judgments into a function that can score new responses cheaply. Policy optimization then uses it as the reward signal.
Important limitations
- It reflects the annotators and prompts in its data, including their biases.
- It is trained on a distribution of responses and may be unreliable for outputs far from it.
- A single scalar can hide trade-offs, such as helpfulness versus safety.
Evaluating it
Check accuracy on held-out preference pairs, look at agreement between annotators as a ceiling, and test how the score behaves under length changes, formatting, and other superficial features.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.