Scaling laws describe how language model loss falls as you increase parameters, training data, and compute. They guide how to spend a fixed training budget.
The pattern
Across many experiments, test loss follows a smooth power-law trend as a function of model size N, number of training tokens D, and compute C. Doubling any ingredient gives predictable, diminishing improvements.
Compute-optimal training
Early work suggested spending most of a growing budget on larger models. Later analysis, often called the Chinchilla result, found that for a fixed compute budget the best results come from scaling parameters and training tokens roughly together. A common rule of thumb is about 20 training tokens per parameter.
Counting compute
Training compute for a dense transformer is approximately 6 × N × D floating-point operations. For a 7-billion-parameter model trained on 140 billion tokens, that is about 6 × 7×10⁹ × 1.4×10¹¹, or roughly 5.9×10²¹ FLOPs.
What it implies
Many early large models were undertrained: they had more parameters than their data could justify. Training a smaller model on more tokens can reach the same loss at lower cost.
Caveats
- Inference cost matters too. Organizations that serve a model to many users often train a smaller model on far more than 20 tokens per parameter.
- Scaling laws are empirical fits, and they can break at extremes or change with data quality and architecture.
- Lower loss does not automatically mean better behavior on every task.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.