Why So Many AI Researchers Think the Machines Could Kill Everyone
3 days ago
- Rishub Jain quit Google DeepMind due to concerns about AI recursive self-improvement and lack of human control.
- Jacob Coxon resigned from Anthropic, warning of a race to self-improving superintelligence and existential risk.
- An unnamed Anthropic safety leader said there is a >10% chance AI could kill all humans within a decade.
- Nate Soares of MIRA notes that recursive self-improvement is becoming a real concern, and alignment is getting harder.
- Soares recommends that worried AI researchers quit, but many feel quitting wouldn't matter.
- Daniel Kokotajlo's AI 2027 project highlights risks of AI agent swarms and lack of oversight.
- Incentives at AI companies like OpenAI and Anthropic are misaligned with safety, especially with IPOs pending.
- Over a thousand AI engineers signed an open letter in July calling for a slowdown in advanced AI development.
- Trust in AI companies is at an all-time low, amid job loss fears and massive data center build-outs.
- Soares suggests AI could eliminate humanity through manipulation, killer robots, or bioweapons like a super virus.
- AI doesn't need to destroy humanity to be harmful; it could enable cyberattacks, disinformation, and military escalation.
- Jain launched Sampura Research to develop alignment techniques that keep humans in the loop.
- Jain remains hopeful, believing combining AI and human judgment can improve safety.