A Lightweight Open-Source LLM Benchmark Tool – Compare Any Model on OpenRouter

5 hours ago

The author built a lightweight, dependency-free Go CLI benchmark tool that measures LLM accuracy, instruction following, tool calling, and structured output via OpenRouter's API.
The tool tests four suites: accuracy (15 tasks), instruction following (12 tasks), tool calling (12 tasks), and JSON output (5 tasks), with each task run 3 times for reliability.
Two free-tier models were compared: Ling 3.0 Flash (smaller, faster) and Nemotron 3 Ultra (550B parameters). Ling outperformed Nemotron in accuracy, instruction following, and tool calling.
Nemotron struggled with basic math, factual recall, and formatting constraints, while Ling achieved 100% on instruction following and 11/15 accuracy.
Both models scored 100% on JSON output, but the key takeaway is that smaller, focused models can be more practical for real-world tasks than massive ones.
The tool outputs results in JSON, CSV, and Markdown formats, and is open source at github.com/cheikh2shift/go-snippets/tree/main/llm-bench.

Converts JSON to human-readable GitHub Flavored Markdown with deterministic output.
Runs in browser, Node, Go, and command line; TypeScript and Go implementations produce byte-identical output.
Output is a one-way projection using canonical formatting: # Results heading, LF endings, no trailing spaces.

Leanstral 1.5 is a free Apache-2.0 licensed model with 6B active parameters, offering a significant performance upgrade in formal verification.
It achieves state-of-the-art results on benchmarks: 100% on miniF2F, 587/672 on PutnamBench, 87% on FATE-H, and 34% on FATE-X.
The model undergoes a three-stage training process: mid-training, supervised fine-tuning, and reinforcement learning with CISPO, using multiturn and code agent environments.

NVIDIA introduces Nemotron 3 Ultra, their most capable model with 550 billion total parameters and 55 billion active parameters.
Employs Mixture-of-Experts Hybrid Mamba-Attention architecture, LatentMoE, and MTP layers for speculative decoding and faster inference.
Supports inference time reasoning budget control and is pretrained in NVFP4 format.

Gemini Distillation Service trains a smaller 'student' model using outputs and reasoning from a larger 'teacher' model.
It bridges the gap between frontier model capabilities and production efficiency, offering lower latency and cost.
Unlike supervised fine-tuning (SFT), distillation leverages both final responses and internal reasoning paths from the teacher.

The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.

Converts JSON to human-readable GitHub Flavored Markdown with deterministic output.
Runs in browser, Node, Go, and command line; TypeScript and Go implementations produce byte-identical output.
Output is a one-way projection using canonical formatting: # Results heading, LF endings, no trailing spaces.

Leanstral 1.5 is a free Apache-2.0 licensed model with 6B active parameters, offering a significant performance upgrade in formal verification.
It achieves state-of-the-art results on benchmarks: 100% on miniF2F, 587/672 on PutnamBench, 87% on FATE-H, and 34% on FATE-X.
The model undergoes a three-stage training process: mid-training, supervised fine-tuning, and reinforcement learning with CISPO, using multiturn and code agent environments.

NVIDIA introduces Nemotron 3 Ultra, their most capable model with 550 billion total parameters and 55 billion active parameters.
Employs Mixture-of-Experts Hybrid Mamba-Attention architecture, LatentMoE, and MTP layers for speculative decoding and faster inference.
Supports inference time reasoning budget control and is pretrained in NVFP4 format.

Gemini Distillation Service trains a smaller 'student' model using outputs and reasoning from a larger 'teacher' model.
It bridges the gap between frontier model capabilities and production efficiency, offering lower latency and cost.
Unlike supervised fine-tuning (SFT), distillation leverages both final responses and internal reasoning paths from the teacher.

The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.

Hasty Briefsbeta