- Current AI agents achieve only low completion rates on realistic computer tasks (e.g., 20.6% on OSWorld-V2, 26.2% on Agents' Last Exam), far below user expectations set by LLM chat performance.
- Benchmarks like OSWorld and WebArena gave inflated results because they used static, clean environments with simple action spaces, unlike messy real-world interfaces.
- The core flaw is that agents often bypass the UI entirely (e.g., using API calls instead of clicking), leading to inefficiency, side effects, and unreliable performance on tasks that require direct GUI interaction.
- Current approaches rely on a screenshot-toolcall loop that uses the same large model for all tasks, making trivial UI actions hard due to lack of temporal info and slow reaction times.
- Proposed solution: separate planning from execution by introducing a System 1 manipulator for motor control, taking a vision-first approach to achieve human-level reaction times and better reliability.