local models

Bilingual

My Local LLM Scored 6/6. It Was Wrong Every Time
2 months ago
- The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
- Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
- Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.
- The key realization was the need to separate measurement into three categories: model capacity, harness capability, and product quality.
- A filter-based approach was adopted: quantify with a real benchmark, filter ideas strictly using numbers, and harvest candidates from published research rather than inventing new ones.
- A scorer bug was found that only verified answer format (bare integer) but not correctness; it was fixed with strict numerical comparison.
- A sealed bank of 100 cases across 25 families showed that tool access raised individual case scores but did not increase complete family coverage for weaker models (e.g., 1.2B stuck at 1/25).
- More capable models (8B, Gemma) saw substantial gains in complete task families from orchestration, while weaker models remained brittle.
- Current rules: grade values not shapes, freeze runtime, avoid agent overfitting, track task families not totals, and publish failures.
Artificial Intelligence: Shades of Gray
2 months ago
- AI is a complex topic with both positive and negative aspects; it's neither always fantastic nor always terrible.
- LLMs are changing work but are also overhyped by vested interests, possibly in a bubble.
- Large-scale GenAI has negative environmental impacts, but local models on solar power can be beneficial.
- GenAI concentrates wealth in megacorps but also lowers entry barriers for newcomers.
- Skeptics should engage with AI to form credible arguments; cheerleaders have valid examples like AI aiding rare disease research.
- AI often fails to deliver perfect answers, sometimes wasting time, sometimes saving it.
- A personal example: using a local LLM on a solar-powered GPU to classify 8000 emails, saving time without relying on cloud AI.
- The email classification task was not fully autonomous but provided practical help, reducing inbox clutter by nearly 1000 messages.
- The author advocates for a balanced, tinkerer's perspective rather than extreme cheerleading or doomsaying.
Pushing Local Models With Focus And Polish
2 months ago
- The author strongly desires local models to be competitive with hosted APIs for practical use in coding agents, but the current experience is fragmented and overly complex.
- Key frustration: local model setup requires many choices (inference engine, model, quantization, template, context size) and config tweaks, often leading to poor results or broken functionality.
- Tool parameter streaming is a critical missing feature in local models—without it, users face long dead connections, can't see tool calls as they happen, and lose ability to interrupt early.
- The local stack is fragmented across many engines (llama.cpp, Ollama, LM Studio, etc.), causing users to deal with a chain of small decisions that affect behavior unpredictably.
- The industry moves too fast, focusing on making each new model run rather than polishing one model and stack to perfection; hosted providers don't ship raw weights and expect users to figure out the rest.
- The author proposes picking one winner (DeepSeek V4 Flash) and polishing it thoroughly with a narrow, model-specific inference engine called ds4.c, designed for Macs with 128GB+ RAM.
- ds4.c is a complete package: Metal path, model-specific loading, prompt rendering, KV handling, server API, and tests—no need for MLX or Ollama.
- The pi-ds4 extension embeds ds4.c into the Pi coding agent, offering zero-configuration local inference that treats the local model as a first-class provider.
- The goal is to achieve hosted-like ergonomics for local models, starting with beefed-out Macs, then scaling to more hardware and models, while keeping everything open and improvable.
- The author encourages those with suitable hardware to try pi-ds4 and hopes to build a critical mass that polishes one coding agent experience end-to-end.

local models

My Local LLM Scored 6/6. It Was Wrong Every Time

Artificial Intelligence: Shades of Gray

Pushing Local Models With Focus And Polish