13 days ago
- A bug in conversation history construction caused small models to fail multi-turn tool calling.
- The bug erased the model's own tool calls and results from its history, making follow-up tasks impossible.
- Correcting the history by including tool calls and results restored 100% success rates across all models.
- Frontier models masked the defect by performing well despite incomplete history, hiding the issue.
- Prompt mitigations interacted unpredictably with the broken history, sometimes causing catastrophic failures.
- Auditing conversation histories and adding regression tests are essential to avoid similar bugs.
- Whole-artifact rewrite designs are immune to this failure class as they have no hidden tool channel.
- Published benchmark results were corrected after identifying the bug, showing tools and rewrite at parity.