There Will Be a Scientific Theory of Deep Learning
7 hours ago
- #Learning Mechanics
- #Deep Learning Theory
- #Neural Networks
- A scientific theory of deep learning is emerging, focusing on training dynamics, hidden representations, final weights, and performance.
- Five research strands contribute: solvable idealized settings, tractable limits, simple mathematical laws, hyperparameter theories, and universal behaviors.
- This theory, termed 'learning mechanics,' describes coarse aggregate statistics and emphasizes falsifiable quantitative predictions.
- It complements other perspectives like statistical and information-theoretic approaches, with a potential symbiosis with mechanistic interpretability.
- Common arguments against fundamental theory are addressed, and open directions and beginner advice are provided.