Bilingual

My Local LLM Scored 6/6. It Was Wrong Every Time
9 hours ago
- The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
- Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
- Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.
- The key realization was the need to separate measurement into three categories: model capacity, harness capability, and product quality.
- A filter-based approach was adopted: quantify with a real benchmark, filter ideas strictly using numbers, and harvest candidates from published research rather than inventing new ones.
- A scorer bug was found that only verified answer format (bare integer) but not correctness; it was fixed with strict numerical comparison.
- A sealed bank of 100 cases across 25 families showed that tool access raised individual case scores but did not increase complete family coverage for weaker models (e.g., 1.2B stuck at 1/25).
- More capable models (8B, Gemma) saw substantial gains in complete task families from orchestration, while weaker models remained brittle.
- Current rules: grade values not shapes, freeze runtime, avoid agent overfitting, track task families not totals, and publish failures.
Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090
5 days ago
- The author tested the Qwen 3.6 35B MoE model on an RTX 3090 with 24GB VRAM using a 4-bit quantized version (UD-IQ4_NL_XL) and Llama.cpp.
- With the GPU only and UD-IQ4_NL_XL quant, the model achieved over 120 tokens/second generation and nearly 2,800 tokens/second prompt processing, but context was limited to about 50,000 tokens.
- Offloading 12 layers' FFNs to the CPU reclaimed VRAM for the full 262,144-token context but reduced performance to ~65 tok/s generation and ~600 tok/s prompt processing.
- Compiling Llama.cpp with CUDA improved performance: 140 tok/s generation and over 3,300 tok/s prompt processing with a 89,600-token context when fully on GPU, and 89 tok/s generation with full context when offloading 10 layers.
- The author automated a benchmark sweep over offload layer counts, using a 2,457-token prompt and generating 6,144 tokens, and found that 12 layers offloaded was a sweet spot for Vulkan and 10 for CUDA.
- Memory usage showed VRAM was maxed until the full context was reached, then dropped smoothly, while system RAM rose as layers were offloaded.
- The study highlighted that only 2B of the 3B active parameters are used for actual thinking, with 1B used for embeddings and output head, and that MoE models require all parameters in memory despite only a subset being active per token.
- The article concluded that while the RTX 3090 can run the model, the 24GB VRAM is limiting, and CUDA provides significantly better performance than Vulkan for this model.
Wolfram LLM Benchmarking Project
11 days ago
- The Wolfram LLM Benchmarking Project focuses on evaluating large language models (LLMs) using Wolfram Language for a code generation task.
- The task converts English-language specifications into Wolfram Language code, based on exercises from 'An Elementary Introduction to the Wolfram Language'.
- The project utilizes tools to assess functional correctness, with test cases previously completed by millions of humans online.
- Results and previous versions are accessible in computable form via the Wolfram Data Repository.
- For LLM developers, Wolfram offers access to datasets, tools, and opportunities for LLM inclusion in benchmarking.

Hasty Briefsbeta

llm benchmarking

My Local LLM Scored 6/6. It Was Wrong Every Time

Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090

Wolfram LLM Benchmarking Project