What loss.backward() actually does

18 days ago

Neural network training involves minimizing loss via gradient descent, which requires derivatives for each parameter.
Reverse-mode automatic differentiation (backpropagation) computes gradients efficiently for many inputs relative to one output.
Each operation's local derivative and the chain rule allow gradient computation through multiplication along graph paths.
Backpropagation uses a topological ordering of the computation graph to ensure gradients propagate correctly in reverse order.
The Value structure in microcrad stores data, gradient, and pointers to operands, enabling automatic graph construction during forward pass.
The backward pass seeds the output gradient as 1 and applies derivative rules per node to accumulate gradients into operands.
Reference counting in C manages memory for the computation graph, preventing leaks without affecting backpropagation logic.
Training involves zeroing gradients, backpropagating, updating parameters via gradient descent, and repeating with new data.

Hasty Briefsbeta