Smevals: A small eval suite for evaluating models, prompts, and harnesses

9 hours ago

smevals is a tool for running small eval suites across different model configurations and grading results.
To use: run `uvx smevals docs` to learn the tool, then build an eval suite.
An eval is a collection of tasks, each a specific challenge (e.g., 'Generate an SVG of a pelican riding a bicycle').
Runs and grading are separate: run eval against models (`uvx smevals run`), then grade results (`uvx smevals grade`).
Results can be viewed via a local web server or built as static HTML for hosting.
Key vocabulary: eval, task, config, run, runner, grader, checks, and checkers.