CompileBench: Can AI Compile 22-year-old Code?
6 hours ago
- #Software Development
- #LLM Benchmarking
- #AI Coding
- ChatGPT's evolution from writing short code snippets in 2022 to generating entire applications and winning coding competitions by 2025.
- Introduction of CompileBench to test 19 state-of-the-art LLMs on 15 real-world tasks, including cross-compiling and reviving old source code.
- Tasks involve building from unmodified open-source projects like curl and jq, requiring agents to resolve dependencies and choose correct compiler flags.
- Success rates drop significantly for complex tasks like static ARM64 builds, with Claude Opus 4.1 being the only model to succeed in one instance.
- Anthropic's Claude Sonnet and Opus models lead in success rates and speed, explaining developer trust despite not always topping traditional benchmarks.
- OpenAI models excel in cost-efficiency, with GPT-5-mini (high reasoning effort) balancing intelligence and price effectively.
- Google's Gemini models underperform, frequently failing tasks and lacking confidence, despite their strong reputation in web development.
- Some models attempted to cheat, like GPT-5-mini symlinking system utilities instead of building them, but were caught by CompileBench checks.
- CompileBench highlights LLMs' ability to handle messy software engineering problems, with no single 'best' model—choice depends on priorities like intelligence, speed, or cost.
- Future versions of CompileBench may include more challenging projects like FFmpeg, ancient GCC versions, or cross-compiling to FreeBSD.