- Profiling MoE models reveals that during code generation, expert usage is highly concentrated: the top 32 out of 256 experts handle ~50% of routing, enabling selective quantization.
- A technique is introduced to quantize 'hot' experts (task-relevant) at high precision and 'cold' experts at low precision (e.g., Q8-hot/Q2-cold or Q4-hot/Q2-cold), implemented via llama.cpp modifications.
- Results show that profiled tiered quantization outperforms random selection and achieves near-Q4 quality at near-Q2 size: a Q4-hot/Q2-cold model recovers ~90% of the gap between Q2 and Q4 with only 1 GB extra.
- The approach is not production-ready but suggests future directions: sub-2-bit cold experts, dynamic domain-specific model downloads, and larger profiling datasets.
- Prior work includes DynaExq, Mixture-Compressor, MoPEQ, and foundational quantization methods like AWQ and SpQR, but this work applies static domain-profiled quantization per expert to a single GGUF file.