Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

3 hours ago

AMD MI355X achieves ~952 tok/s/node for Kimi K3, outperforming B200 on performance per dollar due to lower cost and higher HBM capacity.
Kimi K3 is a 2.8T-parameter open-source model that cannot fit on a single B200 node, requiring two nodes or a B300 node with 288GB VRAM per GPU.
AMD MI355X offers comparable specs to B300 at ~2.4× lower cost per GPU, making it a cost-efficient alternative despite software challenges.
Software issues on ROCm were resolved with simple fixes (e.g., adding top-k renorm function for speculative decoding, padding head count for MLA prefill), not requiring custom kernels.
Speculative decoding improved single-stream throughput by ~2.2× and peak aggregate by +18%, while prefill optimizations cut time-to-first-token by 2–3×.
The MI355X's large HBM capacity gives it a practical edge over B200 for models like Kimi K3 that require >1.5TB VRAM for weights and KV cache.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 16 of 896 experts active per token, a 1M-token context window, and native vision, built on Kimi Delta Attention (KDA) and Attention Residuals.
vLLM provides efficient day-0 support for Kimi K3, adapting its architecture with a hybrid KV-cache manager that handles both KDA recurrent state and full-attention paged KV under one scheduler.
The model achieves up to 370 tok/s per user with DSpark speculative decoding (3.14× improvement over 118 tok/s without speculation) on 16 NVIDIA GB300 NVL72 GPUs.

AMD MI355X GPUs offer significantly lower cost per GPU (about 2.75x cheaper) compared to NVIDIA Blackwell B300, providing a cost-effective solution for AI inference.
Wafer achieved 2626 tok/s/node aggregate throughput with GLM5.2 on AMD MI355X at 2.4 RPS, achieving 80% of B200 performance at over 2x lower cost, and 213 tok/s in single-stream decoding.
Optimizations included quantizing GLM-5.2 to lossless MXFP4 with AMD Quark, using sglang as the inference framework, fixing speculative decode issues, and tuning MoE kernels to improve prefill performance.

Kimi K3 achieves 86.4% resolution rate but requires 8×B300 nodes and is slower than Claude Code.
Token costs for AI coding agents are rising sharply, with 90th percentile users spending ~$7,300/year.
Over 70% of API revenue from major model providers comes from coding use cases.

Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.

Kimi K3 ranks 1st on single-shot Frontend Arena with 1392 Elo, a significant jump over previous Kimi models.
It uses over 12x more thinking tokens than Claude Opus 4.8 and double that of Kimi K2.6.
Kimi K3's performance is attributed to a unique chain-of-thought that simulates an AI agent, iterating on designs like an agent would.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 16 of 896 experts active per token, a 1M-token context window, and native vision, built on Kimi Delta Attention (KDA) and Attention Residuals.
vLLM provides efficient day-0 support for Kimi K3, adapting its architecture with a hybrid KV-cache manager that handles both KDA recurrent state and full-attention paged KV under one scheduler.
The model achieves up to 370 tok/s per user with DSpark speculative decoding (3.14× improvement over 118 tok/s without speculation) on 16 NVIDIA GB300 NVL72 GPUs.

AMD MI355X GPUs offer significantly lower cost per GPU (about 2.75x cheaper) compared to NVIDIA Blackwell B300, providing a cost-effective solution for AI inference.
Wafer achieved 2626 tok/s/node aggregate throughput with GLM5.2 on AMD MI355X at 2.4 RPS, achieving 80% of B200 performance at over 2x lower cost, and 213 tok/s in single-stream decoding.
Optimizations included quantizing GLM-5.2 to lossless MXFP4 with AMD Quark, using sglang as the inference framework, fixing speculative decode issues, and tuning MoE kernels to improve prefill performance.

Kimi K3 achieves 86.4% resolution rate but requires 8×B300 nodes and is slower than Claude Code.
Token costs for AI coding agents are rising sharply, with 90th percentile users spending ~$7,300/year.
Over 70% of API revenue from major model providers comes from coding use cases.

Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.

Kimi K3 ranks 1st on single-shot Frontend Arena with 1392 Elo, a significant jump over previous Kimi models.
It uses over 12x more thinking tokens than Claude Opus 4.8 and double that of Kimi K2.6.
Kimi K3's performance is attributed to a unique chain-of-thought that simulates an AI agent, iterating on designs like an agent would.

Hasty Briefsbeta