An Advanced System Architecture Breakdown of OpenAI's Jalapeno Accelerator
3 days ago
- OpenAI presented the Jalapeno chip, a general-purpose AI inference accelerator optimized for frontier model workloads, focusing on end-to-end latency and energy efficiency rather than peak FLOPs.
- The architecture uses a half-flattened bounded 2-hop Clos topology with Broadcom Tomahawk 6 switches to balance tensor and MoE parallelism with predictable latency.
- Jalapeno employs a spatial architecture with tiled cores, dedicated HBM4 memory (6 stacks, 15.4 TB/s, 216 GiB), and consumes 700W, achieving up to 13 PFLOPS fp4.
- The chip is manufactured on TSMC 3nm and integrates speculative decoding with multi-token prediction to overcome memory bandwidth bottlenecks.
- OpenAI leveraged AI-assisted high-level synthesis using Google XLS and agentic coding paradigms to accelerate chip design, though full agentic flows are not yet mainstream.
- A full 2,048-chip cluster delivers 27 EFlops/s, 32 PB/s, and 432 TiB memory, spanning 28-32 racks.
- Key design tradeoffs included balancing memory bandwidth vs capacity, programmability vs efficiency, and addressing latency sources like network hops and compute stalls.