{"id":22998,"date":"2011-05-03T18:24:18","date_gmt":"2011-05-03T18:24:18","guid":{"rendered":"https:\/\/scannn.com\/prime-radiant-inc-smevals-a-framework-for-running-evals-against-small-and-large-models-%c2%b7-github\/"},"modified":"2011-05-03T18:24:18","modified_gmt":"2011-05-03T18:24:18","slug":"prime-radiant-inc-smevals-a-framework-for-running-evals-against-small-and-large-models-%c2%b7-github","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/prime-radiant-inc-smevals-a-framework-for-running-evals-against-small-and-large-models-%c2%b7-github\/","title":{"rendered":"prime-radiant-inc\/smevals: A framework for running evals against small (and large) models \u00b7 GitHub"},"content":{"rendered":"\n
\n

<\/a>
\n\"Changelog\"<\/a>
\n\"Tests\"<\/a>
\n\"License:<\/a><\/p>\n

A framework for running evals against small (and large) models<\/p>\n

Or pip install smevals<\/code>, or just uvx smevals --help<\/code>.<\/p>\n

\n

Vocabulary used by this project<\/h2>\n

<\/svg><\/a><\/div>\n

The top-level concept is an Eval<\/strong>: a collection of Tasks used to determine how good a particular model or model-and-harness configuration is at a specific high-level capability, such as text-to-SQL, drawing a pelican riding a bicycle, or evaluating whether an implementation satisfies a provided specification.<\/p>\n

Evals can optionally be grouped into Suites<\/strong> of related Evals, primarily as a mechanism for organizing them on disk.<\/p>\n

An Eval<\/strong> is a collection of Tasks<\/strong>. These are the individual exercises that a model must complete for its abilities to be evaluated.<\/p>\n

A Config<\/strong> describes the setup used to attempt Tasks. It specifies a model and may include model parameters, system prompts, tools and other settings.<\/p>\n

To gather evidence, we create a Run<\/strong>. A Run is the immutable record of executing one Task against one Config using a Runner<\/strong>. A Runner is a reusable CLI program that may send prompts directly to a model or build on an agent harness such as Codex or Pi.<\/p>\n

The same Task and Config can be executed multiple times, producing multiple Runs to help account for non-deterministic results – smevals run -n 5<\/code> tops each Task up to five Runs. Each Run includes a timestamp to help track these.<\/p>\n

A Run whose Runner exits non-zero is a failed Run<\/strong>: a harness-level error such as a network failure, not evidence about the model. Failed Runs stay on disk for debugging but are never graded, are excluded from reports, and do not count towards -n<\/code> targets.<\/p>\n

Once we have gathered Runs, we apply a Grader<\/strong> to each Run to produce a Grade<\/strong>. A Grader is a configured sequence of Checks<\/strong>, plus rules for combining their results into that Grade.<\/p>\n

Checks are individual assertions or measurements. Some may be simple, such as \u201cdoes the output contain this text?\u201d Others may be more complex, such as \u201crender this SVG to an image and have an LLM judge assess it\u201d. Each Check names the Checker<\/strong> that performs it, along with configuration for that Checker such as patterns, rubrics or expected values. A Check can be marked as required, in which case its failure halts the Grader and skips the remaining Checks.<\/p>\n

A Checker<\/strong> is a named operation or<\/em> a reusable CLI program that implements one kind of Check. contains<\/code> and xml-valid<\/code> are named operations, ..\/checkers\/render-svg<\/code> might be a custom program. The same Checker can be used by many Checks across many Graders. The Checks in a Grader execute in order and share a working directory, so a Checker may create files – such as a rendered image – which are kept with the Grade as artifacts and available to later Checks in the sequence.<\/p>\n

A Grade<\/strong> is the result of applying a Grader to a Run. It records the result of each Check and may contain an overall pass\/fail outcome and\/or a numeric score. Grades can also include additional notes which are not used for scoring but may help interpret the results in the future.<\/p>\n

We may later change the Grader we use to evaluate Runs without executing the Runs again. A single Run can therefore be evaluated multiple times, producing multiple Grades using different Graders.<\/p>\n

An Eval is any directory containing an eval.yaml<\/code> file:<\/p>\n

\n
my-eval\/\n\u251c\u2500\u2500 eval.yaml            # name and description\n\u251c\u2500\u2500 tasks\/               # one YAML file per Task\n\u251c\u2500\u2500 configs\/             # one YAML file per Config\n\u251c\u2500\u2500 graders\/             # one YAML file per Grader\n\u251c\u2500\u2500 checkers\/            # custom Checker executables (by convention)\n\u251c\u2500\u2500 run-llm              # Runner executable (any name, any location)\n\u2514\u2500\u2500 runs\/                # created by smevals run - never edit by hand\n<\/code><\/pre>\n<\/div>\n
\n
Example Eval: Grading Haikus<\/h3>\n
<\/svg><\/a><\/div>\n
Here’s how to structure a complete Eval that asks models to write haikus and grades them on their structure.<\/p>\n
The Eval consists of five files.<\/p>\n
my-eval\/eval.yaml<\/code> defines a name and description:<\/p>\n
\n
name<\/span>: haiku<\/span>\ndescription<\/span>: >-<\/span>\n  Can the model write a haiku on demand? Graded on structure:<\/span>\n  the reply must be exactly three lines.<\/span><\/pre>\n<\/div>\n
An Eval must have one or more Tasks. Each of these is defined as a tasks\/*.yaml<\/code> YAML file.<\/p>\n
my-eval\/tasks\/pelicans.yaml<\/code> must have a name<\/code>; a prompt<\/code> is the common case, but any other keys are allowed and are passed to the Runner as environment variables:<\/p>\n
\n
name<\/span>: pelicans<\/span>\nprompt<\/span>: Write a haiku about pelicans. Reply with only the haiku, three lines.<\/span><\/pre>\n<\/div>\n
An Eval also needs at least one Config, defined in configs\/*.yaml<\/code>. If there is just one of these it should be called default<\/code>.<\/p>\n
my-eval\/configs\/default.yaml<\/code> – the Config named default<\/code> is used when no -c<\/code> option is passed to smevals run<\/code>.<\/p>\n
runner<\/code> specifies a path to an executable program relative to this file:<\/p>\n
\n
name<\/span>: default<\/span>\nrunner<\/span>: ..\/run-llm<\/span>\nmodel<\/span>: gpt-4.1-mini<\/span><\/pre>\n<\/div>\n
Here’s that Runner script:<\/p>\n
my-eval\/run-llm<\/code> – This one uses the llm<\/a> CLI, but any executable honoring the contract below works. Make it executable with chmod +x<\/code>:<\/p>\n
\n
#!<\/span>\/usr\/bin\/env bash<\/span>\nset<\/span> -euo pipefail\n\nllm -m \"<\/span>$SMEVALS_MODEL<\/span>\"<\/span><\/span> \"<\/span>$SMEVALS_PROMPT<\/span>\"<\/span><\/span>\nllm logs -c --json ><\/span> log.json<\/pre>\n<\/div>\n
The Eval also needs a default Grader, which will be used to grade the results of each Run:<\/p>\n
my-eval\/graders\/default.yaml<\/code>:<\/p>\n
\n
name<\/span>: default<\/span>\nchecks<\/span>:\n  - checker<\/span>: ..\/checkers\/three-lines<\/span>\n    required<\/span>: true<\/span>\nscoring<\/span>:\n  pass_threshold<\/span>: 1.0<\/span><\/pre>\n<\/div>\n
The checker<\/code> can be a relative path to a script – similar to runner:<\/code> above – or can be the name of a built-in checker, listed below.<\/p>\n
my-eval\/checkers\/three-lines<\/code> – a custom Checker, also chmod +x<\/code>:<\/p>\n
\n
#!\/usr\/bin\/env python3<\/span>\nimport<\/span> json<\/span>, os<\/span>, pathlib<\/span>, sys<\/span>\n\nraw<\/span> =<\/span> (pathlib<\/span>.Path<\/span>(os<\/span>.environ<\/span>[\"SMEVALS_RUN_DIR\"<\/span>]) \/<\/span> \"output.txt\"<\/span>).read_text<\/span>()\nlines<\/span> =<\/span> [line<\/span> for<\/span> line<\/span> in<\/span> raw<\/span>.strip<\/span>().splitlines<\/span>() if<\/span> line<\/span>.strip<\/span>()]\nprint<\/span>(json<\/span>.dumps<\/span>({\n    \"score\"<\/span>: 1.0<\/span> if<\/span> len<\/span>(lines<\/span>) ==<\/span> 3<\/span> else<\/span> 0.0<\/span>,\n    \"metrics\"<\/span>: {\"line_count\"<\/span>: len<\/span>(lines<\/span>)},\n    \"notes\"<\/span>: f\"{<\/span>len<\/span>(lines<\/span>)}<\/span><\/span> non-empty line(s)\"<\/span>,\n}))\nsys<\/span>.exit<\/span>(0<\/span> if<\/span> len<\/span>(lines<\/span>) ==<\/span> 3<\/span> else<\/span> 1<\/span>)<\/pre>\n<\/div>\n
To run the eval, grade it and then view the results:<\/p>\n
\n
smevals run my-eval -g                 #<\/span> run every task, grade as each finishes<\/span>\nsmevals run my-eval -m gpt-4.1-nano -m gemini-2.5-flash -g   #<\/span> more models<\/span>\nsmevals run my-eval -n 5 -g            #<\/span> top every task up to five graded runs<\/span>\nsmevals report my-eval                 #<\/span> markdown report in the terminal<\/span>\nsmevals serve my-eval                  #<\/span> live web UI on http:\/\/127.0.0.1:7001<\/span><\/pre>\n<\/div>\n
smevals run<\/code> executes the Runner once per Task\/model combination, with no arguments. Everything arrives through environment variables:<\/p>\n