Reinforcement learning for language models starts with a change of viewpoint. A model that predicts the next token can be read as a policy that chooses actions.
The policy view
At each step, the state is the prompt plus the tokens generated so far. The action is the next token. The model's probability distribution over tokens is the policy. A complete response is a trajectory, and generating it is sampling from the policy step by step.
What pretraining optimizes
Maximum-likelihood training teaches the model to imitate text in a dataset. It rewards matching what humans wrote, not producing the best response according to a goal. This makes it imitation, not optimization.
The compounding-error issue
During imitation, the model always sees correct prefixes. At generation time it conditions on its own earlier tokens. A small mistake can steer it into territory it rarely saw in training, and errors can build up.
Introducing reward
Reinforcement learning replaces "match the data" with "maximize a reward". The reward can come from a reward model trained on human preferences, from a verifier that checks an answer, or from a task outcome.
Why the framing helps
Once responses are trajectories and a reward scores them, standard tools apply: policy gradients, baselines, trust regions, and preference objectives. The rest of post-training builds on this bridge.
Continue with the book
Explore the full treatment, worked examples, and exercises in RL for LLM Agents and Reasoning Models.
Adapted from RL for LLM Agents and Reasoning Models, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.