Training classifiers and language models usually means minimizing cross-entropy. The reason becomes clear once you connect it to entropy and KL divergence.
Entropy
The entropy of a distribution P is the expected surprise: H(P) = −Σ P(x) log P(x). It measures the average number of bits (or nats) needed to describe outcomes drawn from P.
Cross-entropy
Cross-entropy H(P, Q) is the expected surprise when outcomes come from P but you encode them using Q: −Σ P(x) log Q(x). It is always at least as large as H(P).
KL divergence
The Kullback–Leibler divergence is the extra cost: KL(P ‖ Q) = H(P, Q) − H(P). It is non-negative and zero only when P and Q match. It is not symmetric, so it is not a true distance.
Why this explains training
When P is the true label distribution and Q is the model's prediction, minimizing cross-entropy over Q is the same as minimizing KL(P ‖ Q), because H(P) does not depend on the model. It is also the same as maximizing likelihood.
Where else it shows up
- Perplexity is the exponential of cross-entropy.
- Knowledge distillation matches a student distribution to a teacher's.
- Reinforcement learning from human feedback often adds a KL penalty to keep a model close to a reference.
A reminder on direction
KL(P ‖ Q) and KL(Q ‖ P) behave differently. The first penalizes Q for missing mass where P has it, while the second penalizes Q for placing mass where P has none.
Continue with the book
Explore the full treatment, worked examples, and exercises in Mathematics for AI and Machine Learning.
Adapted from Mathematics for AI and Machine Learning, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.