Original companion articles exploring the mathematics and engineering of large language models.
← Back to Blog & Articles Training a critic for a large language model is costly and can be unstable. Group-relative methods remove it […]
← Back to Blog & Articles Direct preference optimization removes the separate reward model and reinforcement-learning loop. It does so by rearranging the KL-regularized […]
← Back to Blog & Articles A reward model is only a proxy for what people want. Push a policy hard against that proxy […]
← Back to Blog & Articles It is easier for people to say which of two responses is better than to assign each a […]
← Back to Blog & Articles Proximal policy optimization became the workhorse of reinforcement learning from human feedback because it delivers trust-region-like stability with […]
← Back to Blog & Articles A policy gradient step that looks good on the current data can be disastrous if it changes the […]
← Back to Blog & Articles Advantage tells a policy whether an action was better or worse than expected. Generalized advantage estimation, or GAE, […]
← Back to Blog & Articles The policy gradient theorem lets us improve a policy by sampling, but the raw estimates are noisy. Baselines […]
← Back to Blog & Articles Maximizing reward alone tends to push a language model far from useful behavior. The KL-regularized objective keeps the […]
← Back to Blog & Articles Reinforcement learning for language models starts with a change of viewpoint. A model that predicts the next token […]
