The singular value decomposition, or SVD, takes any matrix and expresses it as a rotation, a stretch, and another rotation. That simple picture underlies compression, dimensionality reduction, and many numerical methods.
The factorization
Every real matrix A can be written as A = UΣVᵀ. The columns of U and V are orthonormal, and Σ is a diagonal matrix whose entries, the singular values, are non-negative and sorted from largest to smallest. Unlike eigendecomposition, the SVD exists for every matrix, including rectangular and singular ones.
What the singular values tell you
Each singular value measures how strongly the matrix stretches space along one direction. The number of non-zero singular values is the rank. A rapid decay suggests the matrix is close to a lower-rank one.
Best low-rank approximation
Keeping only the k largest singular values gives the best rank-k approximation in both the spectral and Frobenius norms. This result, the Eckart–Young theorem, is the foundation for principal component analysis, latent semantic analysis, and image compression.
Where it appears in ML
- PCA: principal components come from the SVD of centered data.
- Pseudoinverse: solving least-squares problems stably.
- Low-rank adaptation: modern fine-tuning methods rely on the idea that useful updates are low rank.
- Conditioning: the ratio of the largest to smallest singular value measures numerical sensitivity.
A useful habit
When a matrix looks large or noisy, inspect its singular values. A sharp drop often reveals the real number of meaningful dimensions.
Continue with the book
Explore the full treatment, worked examples, and exercises in Mathematics for AI and Machine Learning.
Adapted from Mathematics for AI and Machine Learning, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.