The scaling factor in attention is not decorative. It keeps dot products in a range where the softmax remains responsive as vector dimension grows.
What grows with dimension
If query and key components are roughly independent with mean zero and similar variance, their dot product sums d component-wise products. Its variance grows in proportion to d, so its typical magnitude grows like the square root of d.
What softmax does with large scores
Large-magnitude logits make softmax outputs extremely concentrated. In that saturated regime, most probabilities are near zero and gradients can become small or unstable. Dividing by the square root of d approximately normalizes the score scale.
It is a modeling assumption
Real learned vectors are not perfectly independent, but the derivation explains the design pressure. Implementations also subtract the maximum logit before exponentiation for numerical stability and apply masks before normalization.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.