- The author tested the Qwen 3.6 35B MoE model on an RTX 3090 with 24GB VRAM using a 4-bit quantized version (UD-IQ4_NL_XL) and Llama.cpp.
- With the GPU only and UD-IQ4_NL_XL quant, the model achieved over 120 tokens/second generation and nearly 2,800 tokens/second prompt processing, but context was limited to about 50,000 tokens.
- Offloading 12 layers' FFNs to the CPU reclaimed VRAM for the full 262,144-token context but reduced performance to ~65 tok/s generation and ~600 tok/s prompt processing.
- Compiling Llama.cpp with CUDA improved performance: 140 tok/s generation and over 3,300 tok/s prompt processing with a 89,600-token context when fully on GPU, and 89 tok/s generation with full context when offloading 10 layers.
- The author automated a benchmark sweep over offload layer counts, using a 2,457-token prompt and generating 6,144 tokens, and found that 12 layers offloaded was a sweet spot for Vulkan and 10 for CUDA.
- Memory usage showed VRAM was maxed until the full context was reached, then dropped smoothly, while system RAM rose as layers were offloaded.
- The study highlighted that only 2B of the 3B active parameters are used for actual thinking, with 1B used for embeddings and output head, and that MoE models require all parameters in memory despite only a subset being active per token.
- The article concluded that while the RTX 3090 can run the model, the 24GB VRAM is limiting, and CUDA provides significantly better performance than Vulkan for this model.