Hello Deep Learning: Automatic differentiation, autograd - Bert Hubert's writings
a day ago
- The chapter introduces a 5-layer neural network capable of recognizing all 10 handwritten digits with near-perfect accuracy, using 109184 weights and 202 biases.
- Learning involves gradient descent, where derivatives of the error with respect to all parameters are computed and used to adjust parameters by a small learning rate.
- Automatic differentiation (autograd) is a core technique that efficiently computes derivatives by building a directed acyclic graph (DAG) from operations, then traversing it topologically to propagate gradients.
- The implementation relies on operator overloading and reference counting (e.g., in C++) to lazily construct the DAG, enabling forward and backward passes for training.
- Key components include the Tensor class with Eigen for fast matrix operations, and methods like 'assureValue' for forward pass and 'doGrad' for backpropagation.