Direct Preference Optimization turns paired preferences into a classification-like objective. It increases the relative probability of a preferred response while controlling drift from a reference model.
The training signal
Each example contains a prompt, a chosen response, and a rejected response. The objective compares how strongly the current policy favors the chosen response relative to how strongly the reference policy favors it.
Why the reference matters
Without a reference, optimization could improve the pairwise objective by moving too far from the pretrained behavior. The reference supplies an implicit regularizing anchor, while a temperature-like parameter controls the strength of the preference separation.
Limits remain
DPO inherits the quality and coverage of the preference data. Inconsistent labels, narrow prompts, length bias, and distribution shift can produce misleading gains. Evaluation should include held-out preferences, task performance, safety checks, and measures of unwanted behavioral drift.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.