A language model does not receive words directly. It receives token identifiers produced by a tokenizer, and that representation shapes sequence length, cost, and the patterns available to learn.
Vocabulary versus sequence length
A larger vocabulary can represent common strings with fewer tokens but requires a larger embedding and output layer. A smaller vocabulary reuses pieces more widely but produces longer sequences. The best trade-off depends on languages, domains, and compute limits.
Uneven treatment of text
Names, source code, rare scripts, and morphologically rich languages may split into many pieces. Two sentences with similar character length can therefore consume very different context. Token-level metrics should be complemented with character-, byte-, or task-level analysis.
Reproducibility requires the tokenizer
Model weights alone are not enough. Normalization rules, special-token assignments, vocabulary files, and pre-tokenization behavior must match training. Changing them changes the input distribution even when the visible text looks identical.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.