A reward model is only a proxy for what people want. Push a policy hard against that proxy and the proxy score keeps rising while true quality falls.
Goodhart's law in practice
When a measure becomes a target, it stops being a good measure. The policy finds responses that exploit quirks of the reward model: excessive length, flattery, particular phrasing, or confident-sounding filler.
What the curve looks like
Plot the proxy reward against the amount of optimization, measured by KL from the reference. The proxy keeps increasing. The gold reward, or human judgment, rises at first, peaks, and then declines.
Why it happens
The reward model has finite data and capacity. It is accurate near the responses it was trained on and unreliable elsewhere. Optimization pushes the policy into the unreliable regions.
Mitigations
- Use a KL penalty or early stopping to limit how far the policy drifts.
- Train larger or ensembled reward models, and use uncertainty estimates.
- Refresh preference data with samples from the current policy.
- Control known biases, such as length.
- Evaluate with independent measures, including human review.
A good habit
Never judge progress by the optimized reward alone. Keep a separate evaluation signal that the policy has not been trained against.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.