Training a critic for a large language model is costly and can be unstable. Group-relative methods remove it by using several samples per prompt as a baseline.
The setup
For each prompt, sample a group of responses and score each one, for example with a verifier that checks correctness. The advantage of a response is computed relative to the others in its group.
Leave-one-out and group-mean baselines
A leave-one-out baseline uses the average score of the other responses in the group, so the baseline never depends on the response it scores. The group mean includes the response itself, which introduces a small bias that shrinks as the group gets larger.
Normalizing
Group-relative policy optimization, often called GRPO, divides the centered scores by the group's standard deviation. This puts prompts of different difficulty on a comparable scale. The advantage is then used in a PPO-style clipped objective, with a KL penalty to a reference model.
Where it works well
It suits tasks with automatically checkable rewards, such as math and code, where the reward is clear and many samples are cheap.
Failure modes to watch
- Length bias: per-sequence versus per-token normalization can reward longer or shorter outputs unintentionally.
- Uniform groups: if every sample gets the same score, the advantage is zero and the prompt teaches nothing.
- Standard-deviation scaling can overweight very easy or very hard prompts.
The tradeoff
You pay in extra samples per prompt, but you save the memory and complexity of a critic.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.