Brood War Bench
7 hours ago
- Codex Astra is the clear leader, achieving 100% win rate at the highest effort setting and consistently outperforming all others.
- All models played at a beginner level or below; none could execute complex strategies, defend simple attacks, or build coordinated armies.
- Grok models were incapable of playing effectively, often generating long reasoning chains without issuing commands and rarely fielding combat units.
- Older models and some newer ones treated Brood War as a turn-based game, leading to defeat while thinking; newer models were more aware of action costs.
- Codex frequently used disruptive tactics like sending early workers to attack, which caused opposing agents to waste time thinking instead of acting.
- Claude Fable showed the most earnest attempt at playing, building economies and climbing tech trees, but still fell short of competitive play.
- The benchmark is not exhausted and presents opportunities for future improvement in agent learning and strategy execution.