Modern training relies on a small family of optimizers. Each one fixes a specific weakness of the one before it.
Stochastic gradient descent
SGD updates parameters using the gradient of the loss on a small random batch. Each step is cheap and noisy. The noise can help escape saddles, but a single learning rate for every direction makes progress slow in narrow valleys.
Momentum
Momentum keeps an exponentially decaying average of past gradients and moves along it. Consistent directions build up speed, while oscillating directions cancel out. This smooths the path and speeds convergence on ill-conditioned problems.
Adaptive methods
AdaGrad scales each parameter's step by the inverse square root of its accumulated squared gradients, which helps with sparse features. RMSProp replaces the accumulation with a moving average so the step size does not decay to zero. Adam combines momentum with RMSProp-style scaling and includes bias correction for the early steps.
Practical notes
- Learning-rate schedules and warmup often matter as much as the optimizer.
- Adam is a strong default for many deep-learning tasks, though tuned SGD with momentum can generalize well in some vision problems.
- Weight decay should be implemented carefully with adaptive methods, which is why decoupled weight decay is common.
How to choose
Start with a well-established default, then adjust the learning rate first. Change the optimizer only when a clear problem, such as unstable or slow training, justifies it.
Continue with the book
Explore the full treatment, worked examples, and exercises in Mathematics for AI and Machine Learning.
Adapted from Mathematics for AI and Machine Learning, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.