Maximizing reward alone tends to push a language model far from useful behavior. The KL-regularized objective keeps the policy anchored.
The objective
Instead of maximizing expected reward only, maximize expected reward minus β times the KL divergence between the new policy and a reference policy, usually the supervised model you started from. The coefficient β sets how strongly the policy is pulled back.
Why anchor to a reference
- Reward models are imperfect. A policy can exploit their blind spots to get high scores with poor responses.
- Fluency matters. The reference model carries language ability that unconstrained updates can destroy.
- Diversity. The penalty discourages collapse onto a few high-reward outputs.
A useful closed form
For a fixed reward function, the optimal policy under this objective has a simple form: the reference policy reweighted by exp(reward/β) and normalized. That relation is the starting point for direct preference optimization, which derives a loss directly from it.
Choosing β
A small β lets the policy move far and risks over-optimization. A large β keeps it near the reference but limits gains. In practice, teams monitor the KL value during training and adjust β or stop early.
In implementation
The KL term is often added per token as a penalty in the reward, using the log-probability ratio between the current and reference models.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.