Pushing Local Models With Focus And Polish
12 hours ago
- The author strongly desires local models to be competitive with hosted APIs for practical use in coding agents, but the current experience is fragmented and overly complex.
- Key frustration: local model setup requires many choices (inference engine, model, quantization, template, context size) and config tweaks, often leading to poor results or broken functionality.
- Tool parameter streaming is a critical missing feature in local models—without it, users face long dead connections, can't see tool calls as they happen, and lose ability to interrupt early.
- The local stack is fragmented across many engines (llama.cpp, Ollama, LM Studio, etc.), causing users to deal with a chain of small decisions that affect behavior unpredictably.
- The industry moves too fast, focusing on making each new model run rather than polishing one model and stack to perfection; hosted providers don't ship raw weights and expect users to figure out the rest.
- The author proposes picking one winner (DeepSeek V4 Flash) and polishing it thoroughly with a narrow, model-specific inference engine called ds4.c, designed for Macs with 128GB+ RAM.
- ds4.c is a complete package: Metal path, model-specific loading, prompt rendering, KV handling, server API, and tests—no need for MLX or Ollama.
- The pi-ds4 extension embeds ds4.c into the Pi coding agent, offering zero-configuration local inference that treats the local model as a first-class provider.
- The goal is to achieve hosted-like ergonomics for local models, starting with beefed-out Macs, then scaling to more hardware and models, while keeping everything open and improvable.
- The author encourages those with suitable hardware to try pi-ds4 and hopes to build a critical mass that polishes one coding agent experience end-to-end.