Benchmarking coding agents on Databricks' multi-million line codebase

12 hours ago

Databricks developed an internal coding benchmark using real engineering tasks from their codebase, evaluating AI coding agents on performance and cost.
The benchmark identified three distinct capability tiers among models, indicating that high-cost models aren't always necessary for common tasks, with models like GLM 5.2 offering competitive quality at lower costs.
Token costs often poorly predict overall task expenses due to variations in reasoning efficiency, with differences in harness performance highlighting the importance of context management.
The team emphasized the need for custom benchmarks over public ones like SWE-Bench to accurately reflect internal needs and ensure optimizations don't hinder developers.
Using merged pull requests as a foundation, tasks were rigorously filtered and specified to create a reliable benchmark, with manual validation to avoid issues like leaked solutions from git history.
Findings support using cost-effective models for routine tasks and intelligent routing to balance efficiency and capability, with plans to expand the benchmark and automate model selection.

Hasty Briefsbeta