MHC: Manifold-Constrained Hyper-Connections
4 months ago
- #Model Scalability
- #Machine Learning
- #Neural Networks
- Proposes Manifold-Constrained Hyper-Connections (mHC) to address challenges in Hyper-Connections (HC).
- mHC projects residual connection space onto a manifold to restore identity mapping property.
- Includes infrastructure optimization for efficiency, improving performance and scalability.
- Empirical experiments show mHC is effective for large-scale training.
- mHC offers a flexible and practical extension of HC for foundational models.