Deep transformers need normalization to train stably. LayerNorm was the original choice. Many modern models use RMSNorm, which is simpler and cheaper.
Why normalize
Without normalization, activations can drift to very large or very small scales as they pass through many layers, which makes gradients unstable. Normalizing each token's activation vector keeps values in a predictable range.
LayerNorm
LayerNorm computes the mean and standard deviation across the feature dimension of a single token. It subtracts the mean, divides by the standard deviation, and then applies a learned scale (γ) and a learned shift (β).
RMSNorm
RMSNorm skips the mean. It divides each vector by its root mean square and applies only a learned scale. There is no centering step and no shift parameter. The result is a smaller computation with fewer parameters, and in practice it often performs about as well as LayerNorm.
Where the norm goes
Placement matters. In post-norm blocks, normalization comes after the residual addition. In pre-norm blocks, it comes before the attention or feed-forward sublayer, leaving the residual path untouched. Pre-norm usually trains more reliably in deep stacks because gradients can flow along the clean residual path.
Choosing
Use RMSNorm when efficiency matters and you follow a recent architecture recipe. Use LayerNorm when matching an existing model or reproducing a published result. Either way, verify the implementation against a reference.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.