Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
8 hours ago
- #Legacy Hardware
- #AI Inference
- #Open Source
- Author successfully runs Google's Gemma 4 26B model on a 13-year-old dual Xeon server with no GPU, achieving about 5 tokens per second decoding speed.
- The project required patching the ik_llama.cpp fork to work on pre-AVX2 CPUs, with Claude AI assisting in diagnosing and fixing silent kernel fall-through bugs in the model's graph ops.
- The fix involves compile-time adjustments and runtime workarounds to handle missing AVX2 instructions, enabling older hardware to run modern MoE models cost-effectively as a local fallback.