10 hours ago
- The 21st-century scientific literature is considered toxic for training useful LLMs due to widespread dishonesty, half-truths, and gaming of metrics.
- A 2024 study found that removing ArXiv, PhilPapers, and NIH ExPorter from training data improved LLM performance and reduced toxic output, suggesting scientific articles harm model quality.
- Scientific metadata (authorship, citations) has been gamed and is unreliable, while high-profile institutions are not immune to misconduct.
- AI agents lack access to the informal social networks and reputational information that human scientists rely on to verify work.
- Potential solutions include creating AI agents that socialize with scientists, recording informal discussions, or incentivizing gossip, though these raise surveillance concerns.