An Empirical Study of Harness Design for Coding Agents
4 hours ago
- The study evaluates coding harnesses component-wise, varying planning, action space, and context management while keeping the execution loop fixed.
- Across four models on SWE-Bench Verified and Terminal-Bench 2.1, 176 matched settings were tested, covering five context-management strategies, four context-window budgets, and ablations of planning and action space.
- Context management is more valuable with tighter context-window budgets, mainly by preventing context-overflow failures.
- Staging rule-based elision before LLM-based summarization is the most efficient context-management strategy; making elided content recoverable adds complexity without accuracy gains.
- Planning acts as an accuracy scaffold for weaker models but becomes a cost saver for stronger models, with minimal accuracy change.
- Predefined tools aid models with weaker bash proficiency, while bash-capable models perform well with a bash-only interface, significantly reducing cost, especially on command-line-centric tasks.
- Trajectory analysis shows: context management extends trajectories without altering behavior much, planning changes where trajectories stop, and action space affects code-writing granularity.
- Findings inform model- and budget-aware harness design and propose a modular evaluation framework for future harness components.