2 months ago
- Neural network training involves minimizing loss via gradient descent, which requires derivatives for each parameter.
- Reverse-mode automatic differentiation (backpropagation) computes gradients efficiently for many inputs relative to one output.
- Each operation's local derivative and the chain rule allow gradient computation through multiplication along graph paths.
- Backpropagation uses a topological ordering of the computation graph to ensure gradients propagate correctly in reverse order.
- The Value structure in microcrad stores data, gradient, and pointers to operands, enabling automatic graph construction during forward pass.
- The backward pass seeds the output gradient as 1 and applies derivative rules per node to accumulate gradients into operands.
- Reference counting in C manages memory for the computation graph, preventing leaks without affecting backpropagation logic.
- Training involves zeroing gradients, backpropagating, updating parameters via gradient descent, and repeating with new data.