The Bitter Lesson
16 hours ago
- #Computation
- #AI Research
- #Bitter Lesson
- General methods leveraging computation are the most effective in AI, as shown by 70 years of research.
- Moore's Law and exponentially falling computation costs favor scalable methods over human knowledge-based approaches.
- AI researchers often focus on human knowledge for short-term gains, but long-term progress relies on computation.
- Examples include computer chess and Go, where search and learning outperformed human knowledge-based strategies.
- In speech recognition, statistical methods (like HMMs and deep learning) surpassed human knowledge-based approaches.
- Computer vision shifted from edge detection and SIFT features to deep learning with convolutions and invariances.
- The bitter lesson: Building human-like thinking into AI agents plateaus progress; breakthroughs come from scalable computation.
- Search and learning are the two most scalable methods in AI, capable of handling arbitrary complexity.
- Minds are irredeemably complex; AI should focus on meta-methods that discover approximations rather than hardcoding knowledge.