Bilingual

Popping the GPU Bubble
6 days ago
- Pipelined decoding overlaps GPU and CPU work to avoid GPU bubbles, where the GPU waits for CPU housekeeping.
- Three mechanisms ensure safety: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie refcounting manages finished requests.
- Performance gains depend on GPU speed and batch size, with up to 35% improvement on faster hardware as bubbles become more significant.
AI inference is obviously profitable
2 days ago
- The belief that AI inference is unprofitable due to high costs is disputed, as evidence suggests it can be profitable.
- Rough cost estimates indicate inference costs around $1 per million tokens, with GPT-5.4-mini charging $4.50, implying high profit margins.
- Open LLMs like DeepSeek report over 80% profit margins on inference, with market costs near $0.87 per million tokens, supporting profitability.
- AI labs like OpenAI and Anthropic may use high inference margins to subsidize training costs, explaining their higher prices.
- Even if major AI labs fail, inference providers without training costs can profitably serve models, showing the inference business is sustainable.
GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell
2 days ago
- AMD MI355X GPUs offer significantly lower cost per GPU (about 2.75x cheaper) compared to NVIDIA Blackwell B300, providing a cost-effective solution for AI inference.
- Wafer achieved 2626 tok/s/node aggregate throughput with GLM5.2 on AMD MI355X at 2.4 RPS, achieving 80% of B200 performance at over 2x lower cost, and 213 tok/s in single-stream decoding.
- Optimizations included quantizing GLM-5.2 to lossless MXFP4 with AMD Quark, using sglang as the inference framework, fixing speculative decode issues, and tuning MoE kernels to improve prefill performance.
- AMD faces challenges with day-0 support and software friction for frontier models, but the performance gap is narrowing as kernel and model optimizations improve, reducing reliance on custom kernels.
- The study highlights that achieving state-of-the-art performance on AMD is increasingly about support and optimization rather than inherent software limitations, indicating erosion of NVIDIA's CUDA advantage.

Hasty Briefsbeta

#ai inference

Popping the GPU Bubble

AI inference is obviously profitable

GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell