Theories of Deep Learning
5 hours ago
- #deep learning
- #theoretical frameworks
- #generalization
- Deep learning has seen exponential empirical success, but theory lagged behind; however, the gap is narrowing with multiple emerging theories.
- Theories in deep learning are categorized into architecture theory, optimization theory, and functional theory, each focusing on different aspects like model design, optimizer behavior, and generalization.
- Categorical Deep Learning uses category theory to generalize all neural network architectures, providing a bridge between constraints and implementations via monads in a 2-category.
- Modular Duality offers an optimization framework based on norms to improve training efficiency, introducing duality maps to align gradient updates with parameter spaces.
- A Theory of Generalization shifts focus from parameter space to output space, using the empirical neural tangent kernel to explain phenomena like benign overfitting, double descent, implicit bias, and grokking.
- Other research areas include mechanistic interpretability and various explanatory principles, but the highlighted theories provide strong foundational frameworks.