Tutorial: Algebraic Foundations Powering FlashAttention

7 hours ago

FlashAttention's efficiency arises from the algebraic property of associativity, enabling parallel reduction over attention computations.
The online softmax algorithm, a key component of FlashAttention, is shown to be associative by using a carrier state (m, d) for running max and denominator, with a merge operation that maintains numerical stability.
Welford's variance algorithm is another example of an associative operation, where a carrier state (count, mean, sum of squared deviations) allows stable streaming computation of variance.
FlashAttention's state includes (m, d, o) for max, denominator, and unnormalized output, with a merge that rescales based on max differences, proving associativity through transport of structure.
Transport of structure demonstrates that any bijection applied to a monoid preserves associativity, allowing generalization of associative reductions in machine learning, such as using base-2 exponentials for efficiency.
Numerical stability analysis ensures no overflow and minimal precision loss in FlashAttention, with error bounds scaling with the number of additions, unaffected by tiling.
The Third Homomorphism Theorem provides a method to check if a sequential loop is secretly associative by verifying existence of forward and backward folds, enabling parallelization.

Hasty Briefsbeta