Low-rank adaptation, or LoRA, keeps a pretrained weight matrix fixed and learns a compact update expressed as the product of two smaller matrices.
The parameter reduction
For a weight with shape m by n, full fine-tuning changes mn values. A rank-r update uses factors with roughly r times m plus n values. When r is much smaller than either dimension, the trainable parameter count drops sharply.
Why it can work
The method assumes that the task-specific change can be represented in a low-dimensional subspace even though the base model is large. The update is scaled and added during the forward pass; it can later be merged into the base weight for deployment.
Practical choices
Results depend on which modules receive adapters, the rank, scaling, dropout, data quality, and optimization settings. LoRA reduces trainable parameters and optimizer memory, but it does not eliminate the cost of running the frozen model itself.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.