Differentiable Programming from Scratch
a year ago
- #automatic-differentiation
- #differentiable-programming
- #optimization
- Differentiable programming is a hot research topic, useful in machine learning and other fields like computer graphics.
- The article explains differentiation from basic calculus to higher dimensions, emphasizing gradients and directional derivatives.
- Optimization via gradient descent is discussed, including challenges like local minima and divergence.
- Automatic differentiation (autodiff) is introduced, with forward and backward modes explained in detail.
- Implementation examples show how to compute derivatives using dual numbers and computational graphs.
- A practical example of de-blurring an image demonstrates applying differentiable programming to solve optimization problems.