Fixed three bugs that made Qwen3.5-122B a daily driver on Mac Studio
7 hours ago
- #Mac Studio Performance
- #AI Inference Optimization
- #Local Large Language Models
- Author switches from DS4 Flash to Qwen 3.5 122B model due to high prefill latency (3-5 minutes for 50k tokens) impacting pair programming workflow.
- qMLX is a fork of rapid-mlx, specialized for hybrid Qwen models on Apple Silicon, adding disk KV restore, eviction, and metrics.
- Three bugs were fixed: removing unique timestamps from system prompts to ensure cache hits, persisting streamed replies on interrupts, and fixing checkpoint eviction to prioritize matchable entries.
- After fixes, cache hits increased dramatically; prefill times dropped from minutes to sub-second for repeated prompts, improving conversational throughput.
- Hardware used: Mac Studio with M3 Ultra, 96GB unified memory; decode speeds range from 55 tok/s (short context) to 28 tok/s (at 64k tokens).
- Honest metrics are emphasized, separating prefill and decode phases to avoid misleading throughput numbers; conversational throughput aligns with decode rate.
- qMLX is available on GitHub, includes benchmark tools, and is optimized for long-context agentic coding on Mac hardware.