A little tool to visualise MoE expert routing
a day ago
- Built a tool to visualize Mixture of Experts (MoE) expert routing during token generation, showing which experts fire at each layer.
- Uses modified llama.cpp codebase (with Claude Code) to output profiling data; top panel shows routing as token is generated, bottom panel shows cumulative heatmap.
- Surprising finding: for any given prompt, ~25% of experts never activate, but the dormant set changes with different prompts.
- Recommends exploring llama.cpp codebase with coding agents to learn about LLM inference internals.