The bitter lesson of browser agents
10 hours ago
- Browser Use initially launched with GPT-4o, which wasn't trained for computer use, so we had to define state and actions manually.
- We moved from predefined state and actions to code-based actions as models improved at coding, allowing the model to write JavaScript for actions.
- Replacing fixed actions with code reduced token usage by 60% for Opus 4.8 and 66% for Kimi K3, and both models solved all 18/18 runs.
- Predefined state limits were the next issue, as elements like cookie buttons could be missed if not in the accessibility tree.
- We removed predefined state by giving the LLM direct CDP access, letting it write code to inspect the DOM or take screenshots as needed.
- CDP was chosen over Playwright because it's Chrome's native protocol and can inspect closed shadow roots directly.
- We reused existing agent harnesses like Pi, Codex, and OpenCode to handle the agent loop, avoiding bugs in the loop.
- The bitter lesson: favor general methods that improve with computation, like reusing proven harnesses, exposing simple interfaces, and letting the model choose observations and actions.