Efficient LLM:Bandwidth, Compute, Synchronization, and Capacity are all you need
9 hours ago
- #LLM Inference
- #Performance Modeling
- #Hardware Architecture
- The paper presents a limit study of transformer-based large language model (LLM) inference, focusing on performance bottlenecks like memory bandwidth, capacity, and synchronization overhead.
- A hardware-agnostic performance model is developed to analyze current and future hardware technologies, including HBM3, HBM4, 3D-stacked DRAM, SRAM-based designs, and distributed clusters.
- Key findings include the need for 100s of GB per server for LLM serving, the importance of high memory bandwidth for user throughput, and synchronization latencies needing to be around 1us to maintain efficiency.
- DRAM-based designs show a fundamental advantage in system-level efficiency, and hardware designs can achieve 2000+ user tokens/sec, but reaching 10,000+ tokens/sec requires algorithmic advances.
- The study provides insights into LLM inference performance limits, guiding optimization of deployment strategies and highlighting benefits of future hardware advancements.