Smevals: A small eval suite for evaluating models, prompts, and harnesses

9 hours ago

smevals is a tool for running small eval suites across different model configurations and grading results.
To use: run `uvx smevals docs` to learn the tool, then build an eval suite.
An eval is a collection of tasks, each a specific challenge (e.g., 'Generate an SVG of a pelican riding a bicycle').
Runs and grading are separate: run eval against models (`uvx smevals run`), then grade results (`uvx smevals grade`).
Results can be viewed via a local web server or built as static HTML for hosting.
Key vocabulary: eval, task, config, run, runner, grader, checks, and checkers.

AI agents are non-deterministic, meaning the same input can produce different outputs, sometimes with negative consequences.
Evaluations are essential for monitoring AI system performance, involving two main patterns: offline and online evals.
Offline evals use a fixed dataset before deployment, like unit tests, to catch regressions in controlled cases.

CueBench is a platform for evaluating and improving AI fluency among developers.
It analyzes sessions with AI coding assistants like Claude Code, Cursor, and Codex, providing scores and insights.
Key metrics include overall scores, risk signals (e.g., session loops), cost & efficiency, and model usage.

Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.

Define quality measurement, set a baseline, and build an automated evaluation pipeline without relying on human judgment.
Use offline evaluation with a static dataset and rubric-based scoring (e.g., LLM-as-judge) to test changes locally.
Include safety-critical inputs like canary or honeypot prompts to catch regressions, evaluated with pass/fail annotations.

Gemini Distillation Service trains a smaller 'student' model using outputs and reasoning from a larger 'teacher' model.
It bridges the gap between frontier model capabilities and production efficiency, offering lower latency and cost.
Unlike supervised fine-tuning (SFT), distillation leverages both final responses and internal reasoning paths from the teacher.

AI agents are non-deterministic, meaning the same input can produce different outputs, sometimes with negative consequences.
Evaluations are essential for monitoring AI system performance, involving two main patterns: offline and online evals.
Offline evals use a fixed dataset before deployment, like unit tests, to catch regressions in controlled cases.

CueBench is a platform for evaluating and improving AI fluency among developers.
It analyzes sessions with AI coding assistants like Claude Code, Cursor, and Codex, providing scores and insights.
Key metrics include overall scores, risk signals (e.g., session loops), cost & efficiency, and model usage.

Senior SWE-Bench treats agents like senior engineers by using realistic, natural language instructions instead of over-specified requirements.
It introduces a validation agent that writes behavioral tests to evaluate tasks, adapting to submitted solutions.
Bug tasks are based on tricky user reports requiring runtime investigation, such as debugging logs and reproduction steps.

Define quality measurement, set a baseline, and build an automated evaluation pipeline without relying on human judgment.
Use offline evaluation with a static dataset and rubric-based scoring (e.g., LLM-as-judge) to test changes locally.
Include safety-critical inputs like canary or honeypot prompts to catch regressions, evaluated with pass/fail annotations.

Gemini Distillation Service trains a smaller 'student' model using outputs and reasoning from a larger 'teacher' model.
It bridges the gap between frontier model capabilities and production efficiency, offering lower latency and cost.
Unlike supervised fine-tuning (SFT), distillation leverages both final responses and internal reasoning paths from the teacher.

Hasty Briefsbeta