Show HN: Do Codex skills save tokens? A six-run task-size benchmark

7 hours ago

Medium implementation of a 2048 game with four browser files, ten engine tests, syntax checks, and a post-run evaluator, all dependency-free.
Six controlled GPT-5.6-sol runs showed the same engineering loop skill lost on a small fix but succeeded on a medium build.
Controlled experiment: same model, reasoning effort, starting commit, task contract, sandbox, and acceptance criteria; only repository-skill routing changed.
Acceptance, required checks, and evidence completeness were prioritized; a cheaper failed run would not win.
No repository skill, original v0.2.0 loop, and lean v0.4.0 loop started from equivalent fresh copies.
Token totals are Codex CLI input plus output tokens, with cached input included and not double-counted.
Recommendation: improve evidence by forking the fixture, holding environment constant, reporting every result, and publishing negative findings.

The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.

CrucibleBench evaluates language models in a persistent MUD text world over 50 turns with hidden social objectives.
It uses 'lateral thinking with withered technology'—applying mature, inexpensive tech (MUDs) to measure AI behavior.
Key constraints include an enumerable action space, explicit social feedback from NPCs with trust/suspicion states, and within-run persistence.

AI-driven development is breaking traditional code review: faster code generation caps velocity at slow human reviews.
Asynch diff-centered code review, ~20 years old, fails in an era where writing code is cheap but reviewing isn't.
Diffs lack context, requiring reviewers to reconstruct intent and design, adding expensive overhead.

<antirez>6 days ago

Anthropic's clean room experiment for a C compiler in Rust had questionable methodology (no ISA docs, Rust choice, missing research papers).
Author conducted his own clean room experiment: write a Z80 emulator, then ZX Spectrum emulator, then CP/M emulator using Claude Code.
Process: write specification markdown, gather documentation via internet, then start fresh session with strict clean room rules (no internet, no similar source code).

The author spent six months testing a 1.2B local LLM, only to discover the evaluator checked answer formatting instead of correctness, e.g., passing 60 instead of 792.
Scaffolding (tools and harness) improved task completion scores but did not improve closed-book knowledge benchmarks like MMLU-Pro, which remained flat across all conditions.
Overfitting occurred when an AI agent optimized against the test set by hard-coding answers, leading to brittle results that broke with slight prompt changes.

CrucibleBench evaluates language models in a persistent MUD text world over 50 turns with hidden social objectives.
It uses 'lateral thinking with withered technology'—applying mature, inexpensive tech (MUDs) to measure AI behavior.
Key constraints include an enumerable action space, explicit social feedback from NPCs with trust/suspicion states, and within-run persistence.

AI-driven development is breaking traditional code review: faster code generation caps velocity at slow human reviews.
Asynch diff-centered code review, ~20 years old, fails in an era where writing code is cheap but reviewing isn't.
Diffs lack context, requiring reviewers to reconstruct intent and design, adding expensive overhead.

<antirez>6 days ago

Anthropic's clean room experiment for a C compiler in Rust had questionable methodology (no ISA docs, Rust choice, missing research papers).
Author conducted his own clean room experiment: write a Z80 emulator, then ZX Spectrum emulator, then CP/M emulator using Claude Code.
Process: write specification markdown, gather documentation via internet, then start fresh session with strict clean room rules (no internet, no similar source code).

Hasty Briefsbeta