Where's the "Intelligence Explosion"?
4 hours ago
- Ramez Naam, a noted futurist, is skeptical that Recursive Self-Improvement (RSI) will lead to a rapid intelligence explosion or singularity.
- AI is already aiding its own improvement, but the feedback loop is not strong enough to become self-sustaining; current estimates suggest a 5–10x gap.
- Real-world AI research is significantly harder than benchmarks or forecasts suggest, with OpenAI's internal data showing much shorter autonomous task horizons.
- There are sharp diminishing returns in AI progress, whether from more tokens, experiments, test-time compute, or agent swarms.
- Better models matter more than more copies, but even scaling models shows diminishing returns in training data, compute, and RL.
- Progress is not accelerating; capabilities are rising quickly but steadily, requiring exponentially more resources to maintain pace.
- Better ideas become harder to find over time, as illustrated by Eroom's Law in drug development and diminishing returns in software R&D.
- The self-improvement loop appears to be 2–3% productivity gain per ECI point, far below the 15–19% threshold needed for self-sustaining or runaway RSI.
- Potential accelerants like hardware improvements or economic feedback loops could help, but current data suggests a fast takeoff is unlikely without a major breakthrough.
- More data and better measurement are needed to track whether the loop is strengthening or weakening over time.