Block Number Formats Are Direction Preservers
10 hours ago
- #vector direction
- #block number formats
- #machine learning
- Block number formats in machine learning preserve vector direction, which is crucial for many ML computations.
- These formats represent numbers with limited precision but are tolerated well by large neural networks with little accuracy loss.
- Block formats decouple magnitude and direction representation, approximating each block as direction × magnitude.
- Preserving direction block by block ensures the direction of the whole vector is preserved, as errors do not accumulate catastrophically.
- Core ML operations like stochastic gradient descent and attention mechanisms rely heavily on vector direction.
- Block floating-point formats succeed because they preserve the geometry of vectors, not individual number accuracy.
- Potential extensions include tensor-level scaling, restricted scaling factors, and empirical block sizing guidance.