Are my evals lying to me?
4 hours ago
- #machine-learning-evaluation
- #target-leakage
- #ai-safety
- In 2021, an evaluation of the Epic Sepsis Model showed its performance plummeted in real-world use, highlighting a target leakage issue where the model relied on a feature (antibiotic orders) that leaked diagnostic information.
- Evaluations in machine learning can be misaligned with reality, flattering developers until systems fail in production, especially as large language models face unbounded and shifting input spaces that test sets can't fully capture.
- Using models in production can break evaluations due to feedback loops; for example, fraud models invite adversarial tactics, while recommendation engines create self-reinforcing preferences, invalidating original validation data.
- Generative models like chatbots make defining 'correct' outputs difficult, as metrics may optimize for wrong behaviors (e.g., sycophancy) and lack ground truth data for rare, complex decisions.
- The solution to imperfect evaluations involves moving from static gatekeeping to continuous observatory approaches: embracing portfolios of conflicting instruments, institutionalizing human sampling of high-friction interactions, and scoping blast radii with safety nets.