The Two MMLU Scores: What a Benchmark Name Does Not Fix
2 days ago
- Two MMLU accuracy scores (0.781 and 0.79) from different builds of the same model family are incomparable due to differences in dataset splits, runner implementations, graders, and prompt formats.
- The benchmark name 'MMLU' fixes only the name, leaving many open variables (split, implementation, grader, network access) that can cause accuracy differences of several percentage points or more.
- Comparability is defined by metrological traceability to a common reference (e.g., a shared measurement procedure), not by the numeric value alone.
- The APL AI-Eval profile binds each claim to a content-addressed frame via hash; two frames differing in fields like 'subset' or 'dataset_split' are considered different scopes by canonical byte equality.
- The verifier returns 'incomparable' when frames differ and no applicable bridge is provided; bridges can permit comparison under specific assumptions and declared losses.
- A valid frame (apl-valid) confirms structural correctness but does not guarantee the accuracy of the score or that the declared procedure was executed as specified.