What I learned this week - Can distillation be stopped, Mythos and the cybersecurity equilibrium, Pipeline RL
17 hours ago
- Distillation of frontier AI models is cheap and hard to stop, threatening labs' business models.
- Hiding chain of thought may not prevent distillation, especially for agentic tool use.
- Companies can distill models by using user-approved sequences (e.g., gold diffs) as RL training targets.
- Mythos shows a discontinuity in cybersecurity by combining multiple vulnerabilities for exploits.
- AI improves vulnerability finding more than patching, which requires careful fixes.
- Possible solutions include formal verification, porting C to Rust, and improved patching workflows.
- Private hoarding of cyberattack capabilities raises ethical concerns about centralized control.
- Pipeline RL addresses training inefficiency from variable-length responses via in-flight weight updates.