Mixture-of-Kittens: our open-source MoE megakernel for NVL72s

5 hours ago

Mixture-of-Kittens (MoK) is an open-source MoE training megakernel for NVL72s that fuses all MoE communication and computation into a single deterministic kernel.
MoK addresses the MoE layer bottleneck, which can consume over half of training time, and increased end-to-end tokens per second by 1.41x in production across multiple NVL72 racks.
Key techniques include pull-based dispatch and push-based combine to optimize NVLink bandwidth and reduce signalling overhead, tunable minibatch size for computation-communication overlap, and inter-SM overlapping using dedicated SMs for communication and computation.
Ring token buffers (macrobatches) eliminate CPU-GPU synchronization, and a reversed ring order minimizes forward replay during the backward pass.
MoK supports BF16 and MXFP8 precision, includes MXFP8 quantization fused into dispatch and GEMMs, and computes router weight gradients efficiently.
Benchmarks show MoK is up to 2.37x faster for MXFP8 forward and 1.78x faster for MXFP8 backward compared to the fastest baseline (DeepEP, NCCL, etc.).
The megakernel uses Cluster Launch Control (CLC) for RDMA overlap and software partitioning for deterministic SM allocation.
MoK is fully open-sourced on GitHub and designed to be flexible and easy to modify for other platforms.

The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 16 of 896 experts active per token, a 1M-token context window, and native vision, built on Kimi Delta Attention (KDA) and Attention Residuals.
vLLM provides efficient day-0 support for Kimi K3, adapting its architecture with a hybrid KV-cache manager that handles both KDA recurrent state and full-attention paged KV under one scheduler.
The model achieves up to 370 tok/s per user with DSpark speculative decoding (3.14× improvement over 118 tok/s without speculation) on 16 NVIDIA GB300 NVL72 GPUs.

First open-source training kernels for Minimax Sparse Attention (MSA) in CuTeDSL for Hopper and Blackwell GPUs are introduced.
MSA features blockwise sparsity with 128 blocks using max-pooling, GQA instead of MLA, and group-wise specialization of proxy heads.
Kernel design includes forward proxy attention with top-k sorting and sparse main attention, and a fused backward pass using cached block indices for efficiency.

The core challenge is balancing throughput and stability when using low-precision quantization like NVFP4 in reinforcement learning (RL) for large language models, as sampling and training instabilities compound.
A baseline recipe uses NVFP4 for MoE layers (97% of parameters) with per-token activation scaling to avoid issues like token-dependent quantization and future-to-past leakage.
Gradient stability is improved via dequantized backward passes, aligning backward operands with forward quantization decisions to reduce mismatches and gradient norm spikes.

MI450 GPU introduces new hardware features like TDM, larger LDS, and workgroup clusters that shift the performance bottleneck from compute to memory for attention decode workloads.
Attention decode kernels are optimized using Gluon, a Triton-based DSL with explicit tensor layouts, allowing fine-grained control over generated instructions.
Key optimizations include: optimizing WMMA tensor layouts, using TDM for efficient data loading with direct path, pipelining with triple buffering to overlap memory and computation, and using Split-k with workgroup clusters to boost parallelism.

The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 16 of 896 experts active per token, a 1M-token context window, and native vision, built on Kimi Delta Attention (KDA) and Attention Residuals.
vLLM provides efficient day-0 support for Kimi K3, adapting its architecture with a hybrid KV-cache manager that handles both KDA recurrent state and full-attention paged KV under one scheduler.
The model achieves up to 370 tok/s per user with DSpark speculative decoding (3.14× improvement over 118 tok/s without speculation) on 16 NVIDIA GB300 NVL72 GPUs.

First open-source training kernels for Minimax Sparse Attention (MSA) in CuTeDSL for Hopper and Blackwell GPUs are introduced.
MSA features blockwise sparsity with 128 blocks using max-pooling, GQA instead of MLA, and group-wise specialization of proxy heads.
Kernel design includes forward proxy attention with top-k sorting and sparse main attention, and a fused backward pass using cached block indices for efficiency.

The core challenge is balancing throughput and stability when using low-precision quantization like NVFP4 in reinforcement learning (RL) for large language models, as sampling and training instabilities compound.
A baseline recipe uses NVFP4 for MoE layers (97% of parameters) with per-token activation scaling to avoid issues like token-dependent quantization and future-to-past leakage.
Gradient stability is improved via dequantized backward passes, aligning backward operands with forward quantization decisions to reduce mismatches and gradient norm spikes.

MI450 GPU introduces new hardware features like TDM, larger LDS, and workgroup clusters that shift the performance bottleneck from compute to memory for attention decode workloads.
Attention decode kernels are optimized using Gluon, a Triton-based DSL with explicit tensor layouts, allowing fine-grained control over generated instructions.
Key optimizations include: optimizing WMMA tensor layouts, using TDM for efficient data loading with direct path, pipelining with triple buffering to overlap memory and computation, and using Split-k with workgroup clusters to boost parallelism.

Hasty Briefsbeta