Bilingual

Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0)
2 days ago
- Benchmarks for LLM inference on Apple Silicon were measured using a production fleet with Ollama API, fixed prompt, and 512-token generation budget.
- Results for M4 Mac mini (16 GB, 120 GB/s) include models from 3B to 14B parameters, with generation speeds ranging from 46.7 tok/s (Llama 3.2 3B) to 11.7 tok/s (Qwen 2.5 14B).
- Future benchmarks will cover M4 Pro, M4 Max, and M3 Ultra chips, with performance expected to scale with memory bandwidth.
- Methodology: Ollama 0.31.2 on macOS 15.3.1, default Q4_K_M quantization, 3 recorded runs after warm-up, median speeds reported. Raw data available under CC BY 4.0.
A little tool to visualise MoE expert routing
2 days ago
- Built a tool to visualize Mixture of Experts (MoE) expert routing during token generation, showing which experts fire at each layer.
- Uses modified llama.cpp codebase (with Claude Code) to output profiling data; top panel shows routing as token is generated, bottom panel shows cumulative heatmap.
- Surprising finding: for any given prompt, ~25% of experts never activate, but the dormant set changes with different prompts.
- Recommends exploring llama.cpp codebase with coding agents to learn about LLM inference internals.
Hetzner is working on LLM Inference
6 days ago
- Hetzner is experimenting with LLM inference via an OpenAI-compatible API, but it's still in early stages with no billing, SLA, or production guarantee.
- The only model currently available is Qwen/Qwen3.6-35B-A3B-FP8, a 35B-parameter Mixture-of-Experts model with 3B active parameters, supporting text and images.
- Initial tests show fast performance: 153 ms median time to first token and 224 output tokens per second, though these are not representative of production loads.
- The product's potential lies in Hetzner's ability to leverage its low-cost hardware operations and GPU utilization, but the current GPU lineup lacks large-scale multi-GPU systems needed for bigger models.
- The experiment's future depends on whether Hetzner expands to larger GPU clusters and a broader model catalog, which could make it a serious competitor in inference services.
Nvidia DGX Spark as a daily driver
10 days ago
- The NVIDIA DGX Spark is a compact, power-efficient ARM computer with an NVIDIA GPU (similar to RTX 5070 CUDA cores), 128 GB unified memory, and 4 TB SSD, running Ubuntu 24.04.
- It performs well for general tasks like web browsing and programming, with a Geekbench 6 score comparable to M3 Pro and AMD Ryzen 9 7950X.
- Gaming is decent: can run Crysis Remastered at 3200x1800 with DLSS, and many modern Steam games work via Proton and FEX translation, though performance is about two-thirds of an RTX 5070.
- Local LLM inference has improved significantly with techniques like NVFP4, multi-token prediction, and DFlash, achieving 30-40 tokens per second for a 27B dense model.
- Some software requires tinkering: Darktable needed a fix for OpenCL readback, Rhino 3D works via Wine with AI-generated patches, and Need for Speed Porsche Unleashed runs well with Wine and FEX.
- A custom shelf was designed to add USB ports, an NVMe SSD, and SD card reader using a Hagibis dock, improving connectivity and cooling slightly.
- Overall, the DGX Spark serves as a capable daily driver for gaming, CAD, photography, and LLM inference, with the ability to resolve compatibility issues using AI agents.
Running Gemma4 on Apple Neural Engine
14 days ago
- Apple Neural Engine (ANE) is an efficient computing unit in Apple devices, introduced with A11 in 2017, offering ~19 TFLOPS FP16 performance in A19 Pro (2026).
- ANE is optimized for convolution, not matrix multiplication (matmul), leading to hardware inefficiencies (~1/3 utilization) for LLMs like Gemma 4, requiring workarounds.
- Gemma 4 E2B (2B params, INT4 quantized) and E4B (4B params) were split into 4 CoreML chunks to fit under 1 GB memory on iPhone 17 Pro (8 GB RAM), enabling ANE-only inference.
- Chunking avoids CoreML conversion failures due to graph depth limits (8-10 Attention layers), but introduces dispatch overhead (~2.3 ms per CoreML call, totaling ~9.2 ms baseline).
- Optimizations via MLComputePlan diagnostics achieved 99.78% ANE operation placement for E2B, boosting speed from 11 tok/s to 31 tok/s, with calculations per chunk taking 10-16 ms.
- Quantization experiments showed INT4 works, but INT3 (W3A16) fails with garbled output due to insufficient codebook resolution (8 vs. 16 points) for Transformer weight distributions.
- Anomalies like +196608 offset in argmax outputs revealed ANE's internal 4-parallel argmax structure, highlighting inference challenges from undisclosed hardware behaviors.
- Reducing chunks from 4 to 2 only increased speed to 31.7 tok/s, showing dispatch tax is minor compared to computation costs, with c4 (lm_head) being the heaviest chunk (~30% of time).
- Comparisons: LiteRT-LM on Metal GPU achieves 56.5 tok/s, 1.8x faster than ANE, due to factors like single command buffer, kernel fusion, and native matmul support, which CoreML can't match.
- Future prospects include porting models like Qwen 3.5 (with RNN/attention hybrid) and potential iOS 27 'Core AI' framework opening ANE instruction queue, which could overcome current constraints.
- The project documents failures (e.g., W2A16 gibberish, W8A8 crashes, INT8 KV cache no speed gain) as valuable logs for optimization, emphasizing experiential understanding over specifications.
What happens between entering the prompt and seeing the first word appear
18 days ago
- LLMs generate responses autoregressively during inference, building output one token at a time via sequential forward passes.
- Inference splits into a prefill phase (parallel processing of the prompt to fill the KV cache) and a decode phase (sequential generation of new tokens).
- The KV cache stores computed Key and Value vectors to avoid recomputation, saving significant work but consuming substantial memory, especially with long sequences.
- Decoding strategies like greedy decoding, top-k sampling, top-p (nucleus) sampling, and temperature scaling control token selection to balance predictability and creativity.
- Memory usage for the KV cache depends on model architecture, sequence length, and batch size, with techniques like Grouped Query Attention (GQA) reducing cache size.
- Beam search explores multiple token sequences for optimal outputs, while repetition penalty mitigates repetitive text in longer generations.
- The overall process involves tokenization, prefill, decode with caching, and detokenization to convert token IDs back to text.

Hasty Briefsbeta

llm inference

Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0)

A little tool to visualise MoE expert routing

Hetzner is working on LLM Inference

Nvidia DGX Spark as a daily driver

Running Gemma4 on Apple Neural Engine

What happens between entering the prompt and seeing the first word appear