People often picture training a neural network as rolling into a valley and getting stuck in a bad one. In high dimensions, the more common obstacle is the saddle point.
Classifying critical points
At a point where the gradient is zero, the Hessian decides what kind of point it is. If all eigenvalues are positive, it is a local minimum. If all are negative, a local maximum. If some are positive and some negative, it is a saddle point.
Why saddles are common
For a point to be a local minimum in d dimensions, curvature must be positive in all d directions. If the signs behaved like independent coin flips, the chance of that would shrink exponentially with d. Mixed curvature is far more likely, so most critical points in large models are saddles.
What happens to training
Near a saddle, gradients become small and progress slows. Gradient descent can stall for a long time. Noise in stochastic gradients, and momentum, help the iterates escape along directions of negative curvature.
Important caveats
- This is a heuristic picture. Real loss surfaces have structure that is not random.
- Flat regions and plateaus can slow training even without saddles.
- Good local minima in large networks often have similar loss values, so reaching a perfect global minimum is usually not the issue.
The practical lesson
When training stalls, consider curvature and optimizer choice, not only whether you have reached a "bad minimum".
Continue with the book
Explore the full treatment, worked examples, and exercises in Mathematics for AI and Machine Learning.
Adapted from Mathematics for AI and Machine Learning, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.