The policy gradient theorem lets us improve a policy by sampling, but the raw estimates are noisy. Baselines are the standard way to tame that noise.
The estimator
The gradient of expected reward can be estimated by sampling a trajectory and weighting the gradient of its log-probability by the reward. It is unbiased, which means that on average it points the right way.
Why variance is a problem
If every reward is positive, every sampled action gets pushed up, just by different amounts. The signal is buried in noise, so many samples are needed. Variance also grows with the length of the horizon, since each token's decision is weighted by rewards that depend on many later choices.
Subtracting a baseline
You can subtract any quantity that does not depend on the sampled action from the reward without changing the expected gradient. Using an estimate of the expected reward as a baseline makes good samples positive and poor ones negative, which sharpens the signal.
Which baseline
- A constant average reduces variance a little.
- A learned value function gives a state-dependent baseline and leads to actor-critic methods.
- A group average from other samples for the same prompt avoids needing a critic at all.
The point to remember
Baselines are free in terms of bias but valuable in terms of variance. Much of the design of modern post-training algorithms is about choosing a good one.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.