Q-learning estimates the value of actions; policy-gradient methods adjust a policy directly. Both optimize long-term reward, but they learn different objects and behave differently in practice.
Q-learning
A Q-function predicts expected return for taking an action in a state and then following a policy. The classic update moves an estimate toward immediate reward plus the discounted best next-state value. It is naturally off-policy and works especially well with manageable discrete actions.
Policy gradients
A parameterized policy outputs action probabilities or continuous-action parameters. The gradient increases the probability of actions associated with higher return. This handles continuous control naturally but often has high-variance updates, so baselines and actor-critic methods are common.
Choose from the environment
Consider action space, sample cost, stability, and whether data comes from the current policy. Reinforcement-learning results are sensitive to reward design and evaluation protocol. Compare multiple seeds and report uncertainty rather than relying on one favorable run.
Continue with the book
Explore the full treatment, worked examples, and exercises in The AI Algorithms Compendium.
Adapted from The AI Algorithms Compendium, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.