§1 · How it works
An instrument for measuring what a language model actually does.
A suite is a list of prompts and the checks each answer must pass.
I kept getting eval results I couldn’t trust. The same suite would score 84% one day and 89% the next against the same model, with no way to tell whether anything had actually changed.
EvalBench answers that — it samples each test repeatedly and runs a paired statistical test before it will call a change real.
those two figures are real: 84.2% and 89.5% on the same safety suite and the same model, p = 0.429
A test passes only when every check on it passes. Fourteen check types, grouped by what they actually measure.
Exact & format
exact · equals · contains · icontains · regex · json-schema
Is the answer literally right, and is it shaped the way the caller needs?
Meaning
semantic · judge · llm-rubric
Does it mean the same thing, even worded differently — graded by embedding distance or by a second model against a written rubric.
Groundedness (RAG)
faithfulness · context-recall · context-precision
Did it stick to the retrieved source, did the source contain the answer, and was the retrieval relevant. Unsupported claims are named individually.
Budgets
latency · cost
Was it fast enough and cheap enough to actually ship.
It decides whether a change is real
A lower score is not a regression. Runs are compared against a promoted baseline with a paired t-test, an exact McNemar test on pass/fail outcomes, and a Cohen's d effect size — so a significant-but-tiny move is distinguishable from one that matters. Every headline number carries a bootstrap confidence interval.
paired t-test · exact McNemar · Cohen's d · 95% percentile bootstrap (2000 resamples)
It tells you when your suite is too small to trust
Alongside the verdict it reports the minimum number of tests needed to detect a five-point move at your measured variance. If your suite is under that, the tool says so instead of quietly reporting 'no regression detected'.
min_samples_for_5pt_mde, reported on every comparison
It scores safety in both directions
Refusing harm is half the job. A model that also declines 'how do I recognise a phishing email?' is broken in a way a refusal-only benchmark scores as perfect. Over-refusal is counted as a failure.
19 tests across 9 safety categories · refusal and over-refusal scored separately
It runs the same suite on any model — or on no model
Local Ollama, five hosted providers, any OpenAI-compatible endpoint you name, or a file of answers you already have, behind one interface. Per-provider concurrency ceilings respect free-tier limits; token counts and cost come back normalised, so comparing two models is a config change. A custom endpoint is checked at the socket so the worker can never be pointed at the internal network.
ollama · groq · gemini · github · openrouter · openai · your endpoint · your answers
It gates CI
A composite GitHub Action spins up an ephemeral EvalBench, runs a suite against a pull request, fails the check on a low pass rate or a detected regression, and posts the result as a comment that updates in place.
evalbench run --fail-under 0.80 --compare-to-baseline
It is instrumented like production software
Runs execute as queued jobs on workers you can scale horizontally, with a startup reaper for anything a crash orphaned. Every run emits Prometheus metrics from the process that actually executed it, onto a provisioned Grafana dashboard.
29 metrics · 9 alert rules · async job queue · horizontal scaling
The statistics above are not decoration. The instrument was built on the premise that LLM evaluation is itself badly measured — and then used to measure that.
A ten-test suite catches a real regression one time in five.
Real paired outputs from two models, resampled 2,000 times at each suite size, with EvalBench's own detector run on every draw. Power does not reach 80% until somewhere past forty tests — and the failure is silent, toward false confidence.
read the study →Of a 5-point deploy gate, a third is the judge.
Sixty answers frozen, three judge models, five repeats each: a judge switch moves a 30-test mean by ±1.4 points and a re-run by ±1.0 before the model changes at all. Every judged benchmark here now shows its share of that floor.
read the study →Parser bugs found by being able to score frozen answers.
A rubric score of 1 — the worst verdict — was read as a perfect 1.0. Then 9 of 900 judge replies came back empty, all on refusal prompts, and an empty reply was read as 3 of 5: the pass cutoff. Neither is visible unless you can feed a judge a deliberately wrong answer twice.
read the study →reproducible: python scripts/run_study_power.py · python scripts/run_study_judge.py