Proximal policy optimization became the workhorse of reinforcement learning from human feedback because it delivers trust-region-like stability with first-order tools.
The probability ratio
For a sampled token, compute the ratio of its probability under the new policy to its probability under the policy that generated the data. A ratio of 1 means nothing changed.
The clipped surrogate
PPO multiplies the ratio by the advantage, then clips the ratio to a narrow band, commonly 1 − ε to 1 + ε with ε near 0.2. The objective takes the minimum of the clipped and unclipped terms.
- Positive advantage: the objective stops rewarding increases in probability once the ratio exceeds 1 + ε.
- Negative advantage: it stops rewarding decreases once the ratio falls below 1 − ε.
This removes the incentive to move far from the old policy in a single update.
The full objective
Practical implementations add a value-function loss for the critic and often an entropy bonus. For language models, a KL penalty against a reference policy is usually included too.
Multiple epochs
Because the objective is bounded, the same batch can be reused for several gradient epochs, which makes sampling more efficient.
Watch-outs
PPO has many interacting details: advantage normalization, learning rates, batch sizes, and the quality of the critic. Monitor the clipped fraction and the KL to the reference to see whether updates are healthy.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.