Models train, from gradient descent to Adam
a day ago
- The article explores how different optimizers affect accuracy on a fixed softmax regression model for MNIST digit classification.
- Training is likened to a hiker descending a foggy mountain; each optimizer provides a different step strategy.
- Full-batch gradient descent gives 41.1% accuracy after 3 steps, while stochastic gradient descent achieves 87.6% with many updates.
- Mini-batch (batch size 128) reaches 85.2%, and momentum/Nesterov acceleration improve slightly to 86.7%/87.6%.
- Adaptive methods (AdaGrad, RMSProp, Adam, AdamW) adjust learning rates per parameter, with AdamW decoupling weight decay from gradient updates.