ChatGPT Optimizes Its Agent Loop: Harness, API, and Inference
5 hours ago
- AI agent systems like ChatGPT have three layers: harness, API, and inference, each with distinct optimization techniques.
- The harness layer reduces repeated work via persistent WebSockets, stable prompt prefixes, deferred tool discovery, and Code Mode.
- The API layer improves efficiency by tokenizing only the delta, running safety checks in parallel with inference, and routing to newer CPUs.
- The inference layer optimizes GPU usage with cache-aware routing, KV cache management, speculative decoding, and separating prefill from decode.
- Key lessons include keeping designs simple, using agents to optimize their own stack, and optimizing end-to-end rather than focusing on a single technique.