23 days ago
- Deep learning has seen exponential empirical success, but theory lagged behind; however, the gap is narrowing with multiple emerging theories.
- Theories in deep learning are categorized into architecture theory, optimization theory, and functional theory, each focusing on different aspects like model design, optimizer behavior, and generalization.
- Categorical Deep Learning uses category theory to generalize all neural network architectures, providing a bridge between constraints and implementations via monads in a 2-category.
- Modular Duality offers an optimization framework based on norms to improve training efficiency, introducing duality maps to align gradient updates with parameter spaces.