Gain of Function, Loss of Control
21 hours ago
- #Safety Theatre
- #AI Governance
- #Gain-of-Function Research
- The AI industry's safety measures resemble 'safety theatre', with a focus on AI-generated bioweapon risks, though real danger may lie elsewhere.
- Gain-of-function (GoF) research in virology, despite funding and favorable conditions, has not produced credible pandemic threats, raising questions about its governance.
- GoF research is incentivized by academia's 'publish or perish' culture, being cheap and fast compared to antiviral research, leading to high-risk projects.
- AI capability enhancement parallels GoF in speed and publication rewards, while AI safety/ethics research is slow, uncertain, and poorly incentivized.
- Both fields rely on self-governance by peers, creating risks of oversight failure, as seen in biosafety standards and AI safety policies.
- Dissenting voices in GoF and AI are suppressed through orchestrated consensus or corporate actions like firing ethics teams, prioritizing capability over safety.
- Leaks in AI, such as code or model releases, parallel lab leaks in virology, spreading risks and necessitating adaptation by third parties.
- BSL-4 labs and data centers are expanding rapidly, often in urban areas with minimal regulation, increasing the likelihood of accidental leaks.
- Structural governance failures in GoF and AI prioritize risky capability enhancements over safety, exposing society to catastrophic risks outweighing benefits.
- Addressing these issues requires building mandatory safety infrastructures and reevaluating the value placed on innovation in both pathogens and AI.