Advantage tells a policy whether an action was better or worse than expected. Generalized advantage estimation, or GAE, gives a tunable way to estimate it.
The advantage
The advantage of an action is its expected return minus the value of the state. Positive advantage means the action was better than average, so the policy should make it more likely.
Two extremes
A Monte Carlo estimate uses the full observed return. It is unbiased but noisy. A one-step temporal-difference estimate uses the immediate reward plus the critic's value of the next state. It has low variance, but it inherits any error in the critic, so it is biased.
GAE blends them
GAE forms an exponentially weighted sum of temporal-difference errors, controlled by a parameter λ between 0 and 1. With λ = 0, it equals the one-step estimate. With λ = 1, it matches the Monte Carlo estimate. Values in between trade bias against variance.
In language model training
A critic predicts the expected reward from each partial response. GAE then assigns credit to individual tokens instead of treating the whole response as one decision. That finer signal can speed up learning when rewards arrive only at the end of the response.
Practical advice
Start with commonly used values for λ and the discount, then watch value-function quality. A poorly trained critic can make GAE estimates misleading, no matter how well you tune λ.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.