The Universal Weight Subspace Hypothesis
2 days ago
- #neural-networks
- #machine-learning
- #spectral-analysis
- Deep neural networks trained across diverse tasks exhibit similar low-dimensional parametric subspaces.
- Empirical evidence shows neural networks converge to shared spectral subspaces regardless of initialization, task, or domain.
- Mode-wise spectral analysis of over 1100 models identifies universal subspaces capturing majority variance in few principal directions.
- Spectral decomposition techniques reveal sparse, joint subspaces consistently exploited across diverse tasks and datasets.
- Findings offer insights into intrinsic organization of information within deep networks and implications for model reusability and efficiency.
- Potential to reduce carbon footprint of large-scale neural models through training and inference-efficient algorithms.