- The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
- Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
- Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.
- The key realization was the need to separate measurement into three categories: model capacity, harness capability, and product quality.
- A filter-based approach was adopted: quantify with a real benchmark, filter ideas strictly using numbers, and harvest candidates from published research rather than inventing new ones.
- A scorer bug was found that only verified answer format (bare integer) but not correctness; it was fixed with strict numerical comparison.
- A sealed bank of 100 cases across 25 families showed that tool access raised individual case scores but did not increase complete family coverage for weaker models (e.g., 1.2B stuck at 1/25).
- More capable models (8B, Gemma) saw substantial gains in complete task families from orchestration, while weaker models remained brittle.
- Current rules: grade values not shapes, freeze runtime, avoid agent overfitting, track task families not totals, and publish failures.