Attention by itself has no sense of word order. Rotary position embeddings, or RoPE, add order by rotating query and key vectors so that their dot product depends on relative position.
The problem
Self-attention compares tokens as a set. Shuffle the inputs and, without position information, each token produces the same representation. Positional encodings restore order.
The rotation trick
RoPE splits each query and key vector into pairs of dimensions and treats each pair as a point in a two-dimensional plane. At position m, pair i is rotated by the angle m · θᵢ. The frequencies θᵢ decrease geometrically across pairs, commonly θᵢ = 10000^(−2*i*/d), so some pairs rotate quickly and others slowly.
Why relative position appears
The dot product of a rotated query at position m and a rotated key at position n depends only on the difference m − n. Absolute positions cancel out. The model therefore sees relative distance directly, without any extra learned parameters.
Practical details
- RoPE is applied to queries and keys, not to values.
- It replaces additive position vectors on the token embeddings.
- Extending context beyond the trained length usually needs frequency scaling or interpolation, because angles at unseen positions fall outside what the model learned.
Comparison with alternatives
Sinusoidal encodings add fixed absolute position vectors to embeddings. Learned absolute embeddings use a trainable table with a hard length limit. ALiBi adds a distance-dependent bias directly to attention scores. RoPE sits between them: it encodes absolute position by rotation but yields relative behavior in the score.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.