A Geometric Calculator Inside a Neural Network
2 days ago
- #addition module
- #neural geometry
- #Fourier features
- Researchers discovered a general-purpose addition module in layer 18 of Llama 3.1 8B that operates using circular representations of numbers through Fourier features.
- This addition module handles various cyclic tasks like month and day calculations by performing modular arithmetic in parallel across different moduli circles (e.g., mod-2, mod-5, mod-10).
- The neural network reuses this mechanism across related tasks due to optimization pressures, demonstrating efficient computational reuse.
- Causal evidence, such as steering manipulations on the addition module, confirms its active role in computations, showing these geometric structures are real computational objects.
- Understanding neural representations and computations is crucial for controlling, debugging, and designing better AI models.