Overtraining as the path to human-like AI
4 hours ago
- #neural networks
- #grokking
- #AI scaling
- Gwern proposes that overtraining large models on small datasets could lead to grokking, enabling human-like generalization in AI.
- Grokking is a phenomenon where prolonged training forces models to find simpler, deeper understandings of data, beyond mere memorization.
- Current AI labs focus on training smaller models on vast datasets, which may hinder grokking and limit generalization capabilities.
- Gwern suggests training a massive model (e.g., 100 trillion parameters) on a constrained dataset to encourage deeper learning, though this approach is untested and risky.
- The idea faces technical and political challenges, including high costs and the appearance of failure during training until a breakthrough occurs.