Perplexity is the standard intrinsic metric for language models. It is a simple transformation of cross-entropy, and it has a useful interpretation.
Cross-entropy
For a text, take the probability the model assigned to each correct next token and compute the average of the negative logarithms. That average is the cross-entropy, and it is the same quantity minimized by maximum likelihood training.
Perplexity
Perplexity is the exponential of cross-entropy, using the same logarithm base. It can be read as an effective branching factor: how many equally likely choices the model is, on average, hesitating between at each step. A model that guessed uniformly among V tokens would have perplexity V. Lower is better.
A tiny worked example
Suppose a model assigns probabilities 0.5, 0.25, and 0.125 to three correct tokens. In base 2, the negative logarithms are 1, 2, and 3 bits. The average is 2 bits. The perplexity is 2² = 4, so the model behaves as if choosing among four equally likely options.
Cautions
- Tokenizer dependence: perplexity depends on how text is split into tokens, so compare models only with the same tokenizer and test set. Bits per byte gives a fairer comparison across tokenizers.
- Data leakage: test text that appeared in training makes perplexity look better than it is.
- Limited meaning: a good perplexity does not guarantee factual accuracy, helpfulness, or safety. Use it alongside task-based evaluation.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.