Parameter counting reveals where model memory and compute are concentrated. The reliable method is to count each matrix from its input and output dimensions, then add biases and normalization parameters if present.
Embeddings and attention
A token embedding contains vocabulary size times model width parameters. A standard attention block has query, key, value, and output projections. When all are square model-width matrices, their leading contribution is four times width squared per layer.
The feed-forward block
A two-layer multilayer perceptron expands from model width to an intermediate width and projects back. Its leading count is twice the product of those widths. Gated variants add another input projection and therefore change the familiar shortcut formula.
Avoid hidden assumptions
Check whether input and output embeddings are tied, whether projections include biases, how many key-value heads are used, and which normalization is implemented. State the architecture first; then a parameter total becomes an auditable calculation rather than a memorized estimate.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.