Direct preference optimization removes the separate reward model and reinforcement-learning loop. It does so by rearranging the KL-regularized objective.
The key observation
For a KL-regularized reward objective, the optimal policy equals the reference policy reweighted by exp(reward/β). Rearranging, the reward can be written as β times the log-ratio of the optimal policy to the reference, plus a term that depends only on the prompt.
Plugging into Bradley–Terry
Substitute that expression into the Bradley–Terry preference model. The prompt-dependent term cancels in the difference between chosen and rejected responses. What remains is a loss that depends only on the policy and the reference.
The loss
DPO minimizes the negative log-sigmoid of β times the difference between the chosen and rejected log-ratios. It increases the relative likelihood of preferred responses compared with rejected ones, anchored by the reference model.
Practical effects
- It trains with a simple supervised-style step on preference pairs.
- It avoids sampling from the model during training in its basic form.
- It needs a frozen reference model's log-probabilities.
Known issues
Likelihood of both chosen and rejected responses can fall, a behavior called likelihood displacement. Offline pairs may differ from the current policy's outputs, and variants add regularization, length normalization, or on-policy data to address this.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.