RL for LLM Agents and Reasoning Models
By Choral Software
Language: English
A Mathematical and Engineering Guide, from MDPs to GRPO
Understand why the methods that train modern language-model agents work, and when they fail.
Reinforcement learning is how modern language models are taught to reason, use tools and act over many steps. RL for LLM Agents and Reasoning Models builds the subject from first principles, with the mathematics worked out and the code written from scratch.
It starts with Markov decision processes and the policy gradient theorem, then develops advantage estimation, trust regions, PPO, reward models, DPO and its variants, and group-relative methods such as GRPO. It goes on to verifiable rewards, outcome versus process supervision, test-time compute, agents as partially observable decision processes, multi-turn and long-horizon training, tool use and environment design, self-play, reward hacking and collapse, evaluation and statistics, and the systems problems of RL post-training. The last chapter maps open problems.
What’s inside
- Foundations: language models as policies, Markov decision processes, and the policy gradient theorem.
- Core algorithms: advantage estimation and critics, trust regions, and proximal policy optimization (PPO).
- Preference learning: reward models, direct preference optimization (DPO) and its variants, and critic-free group-relative methods such as GRPO.
- Reasoning and agents: verifiable rewards, outcome versus process supervision, test-time compute, multi-turn and long-horizon training, tool use, environment design, and self-play.
- In practice: reward hacking and collapse, evaluation and statistics, systems problems of RL post-training, and open problems.
- Five appendices: notation, a tested utility module, deferred proofs, a reproducibility checklist, and worked solutions.
Who it is for: readers who know calculus, probability and basic machine learning and want to understand why these methods work and when they fail. All data are synthetic, and the listings run on a laptop CPU.
Features: 21 chapters; solutions to 167 exercises; original Python programs (NumPy, SciPy, PyTorch) on synthetic data with fixed seeds; experiments reported as they came out, including negative and null results.
Availability: Coming soon.