Forget "Fat" Models. The Future Is Narrow
5 hours ago
- #Local Inference
- #MoE Compression
- #AI Models
- The author delays building a local AI inference rig due to fast market changes and high hardware prices.
- China may restrict export of top AI models, including open-weight ones like Qwen and GLM-5.2, threatening Western access.
- Open-weight models from China enable affordable local and serverless AI; a ban could limit this access.
- Mixture-of-Experts (MoE) models use sub-networks (experts) activated per token for efficiency.
- REAP (Router-weighted Expert Activation Pruning) prunes unused experts in MoE models without retraining, based on saliency scores.
- Calibration data is crucial: using task-specific data (e.g., code) preserves performance in that area but reduces generality.
- Pruning half of Qwen3-Coder-480B experts retains ~97% coding ability but worsens general knowledge by ~8%.
- Expert merging is less effective than pruning due to router confusion; removal maintains accuracy.
- Visualization shows pruning focuses on late layers where experts specialize, with early layers more uniform.
- 0xSero applied REAP and quantization to compress DeepSeek-V4-Flash to 103GB, running on a DGX Spark.
- Slight retraining (Router-KD) may be needed post-pruning to fix issues like repetition.
- Future tooling could simplify REAP, enabling narrow models for specific use cases on consumer hardware.
- Combining compression with optimized inference engines could create plug-and-play AI appliances.
- If China restricts models, learning to prune and run local models becomes a strategic advantage.