We removed 80% of our agent's tools

4 months ago

The team built a sophisticated internal text-to-SQL agent, d0, which initially had a fragile and slow performance with an 80% success rate.
They simplified the agent by stripping it down to a single tool: execute arbitrary bash commands, resulting in a 100% success rate, fewer steps, and faster responses.
d0 is designed to translate natural language questions into SQL queries, democratizing data access across the company.
The initial approach involved heavy prompt engineering and multiple specialized tools, which required constant maintenance and re-calibration.
The new approach, dubbed the 'file system agent,' gives Claude direct access to files, allowing it to use Unix tools like grep, cat, and ls to understand and query data.
The new architecture is 3.5x faster, uses 37% fewer tokens, and has a 100% success rate compared to the old system.
Key lessons include not fighting gravity (leveraging existing powerful abstractions like file systems), trusting the model's reasoning, and ensuring good documentation and data structure.
The team emphasizes starting with the simplest possible architecture and adding complexity only when necessary, building for future model improvements.

Hasty Briefsbeta