A policy gradient step that looks good on the current data can be disastrous if it changes the policy too much. Trust-region methods put a limit on that change.
The danger of large steps
The gradient is estimated from samples drawn by the current policy. After a large update, the new policy visits different outputs, and the estimate is no longer reliable. Performance can collapse, and the collapse may be hard to recover from.
Measuring change in policy space
Distance between parameters is a poor measure, since the same parameter shift can change behavior a little or a lot. A better measure is the KL divergence between the old and new policies, which compares what they actually do.
The monotonic-improvement idea
Using the performance-difference lemma, one can build a surrogate objective that approximates the improvement from a new policy, with an error that grows with the KL divergence. Maximizing the surrogate while bounding the KL provides a guarantee that performance does not drop.
Natural gradients
The Fisher information matrix describes the local geometry of the KL divergence. Preconditioning the gradient with it gives the natural gradient, which takes steps of a controlled size in policy space.
Where this leads
Exact trust-region methods are expensive. Proximal policy optimization approximates the same idea with a simple clipped objective that is easier to implement at scale.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.