A Trick for Backpropagation of Linear Transformations
3 days ago
- #einsum
- #backpropagation
- #linear-algebra
- Einsum (Einstein summation) can represent various linear transformations like sums, matrix products, dot products, and Hadamard products.
- A simple trick for backpropagation through einsums involves swapping the letters used in the forward pass to compute gradients.
- For example, in matrix multiplication, swapping the output letters of the forward pass einsum helps derive the gradient with respect to the input matrices.
- The shape of the gradient matches the input matrix because the output letters in the backward pass correspond to the input letters in the forward pass.
- The einsum trick can be interpreted as a matrix multiplication with transposed matrices, simplifying gradient computation.
- Verification using JAX's automatic differentiation confirms the correctness of the derived backpropagation formula.
- Einsums are a powerful tool for representing and reasoning about linear transformations, and the swapping trick simplifies backpropagation.