EvalBench

§ Research · two studies

LLM evaluation is itself badly measured.

Every benchmark on this site reports what it can actually see. These are the two studies, run with EvalBench on real model outputs, that say why it has to: how many tests a regression gate needs before its verdict means anything, and how much of a judged score is the judge.

How many tests does an LLM eval suite need?

An empirical power analysis of regression detection. The short answer is that most eval suites are far too small to gate a deploy on — and they fail in the direction that hurts.

2026-09-07 · 30 tests · baseline openai/gpt-oss-120b · candidate allam-2-7b · 2,000 bootstrap resamples per size

Teams gate deploys on eval suites, and those suites are usually small — a dozen prompts, maybe thirty. When such a suite reports “quality dropped”, how often is that verdict trustworthy? And how many tests does it actually take before a real regression is reliably caught?

1

Collect

A 30-test suite spanning factual recall, arithmetic, multi-step reasoning, structured output and definitions, run once against two models of clearly different capability. Deterministic assertions only — no LLM judge — so the variance measured comes from the model, not the grader.

2

Observe

The real effect was a mean per-test score difference of -0.2302 (0.8378 → 0.6076), with a standard deviation of differences of 0.5478.

3

Resample

For each suite size n, 2,000 bootstrap draws of n test-pairs were taken with replacement and EvalBench’s own detector run on each. The share of draws reporting a regression is the empirical power.

Fig. 1

Probability the regression is detected

0%25%50%75%100%80% power — the conventional barn≈60410203060100tests in the suite (n)

Each point is 2,000 bootstrap draws at that suite size. The dashed line is 80% power, the conventional threshold for calling a test adequately powered.

Power at n=1021%A ten-prompt suite misses this real regression four times in five.
80% power atn≈60Tests needed before the verdict is dependable.
Effect size-0.4202Cohen's d — a moderate effect, not a subtle one.
Observed p0.028729At the full n=30, the drop is significant.
4.1

Small suites miss real regressions

At ten tests the detector fires only 21% of the time on an effect this size. A team running that suite would miss the same genuine degradation four times in five, and reasonably conclude nothing had changed.detail
power(n=10) = 0.2135 · power(n=30) = 0.594 · power(n=60) = 0.9105
4.2

The failure is silent, and in the wrong direction

A small suite rarely invents a regression that isn’t there; it fails to see one that is. For a deploy gate that means the failure mode is ship it, not investigate — the expensive direction to be wrong in.

4.3

The tool's own estimate is a safe upper bound

EvalBench reported that 943 tests would be needed. The measured requirement was about 60. The estimate is conservative rather than wrong — it targets a 5-point shift, and the effect here was much larger.detail
min_samples_for_5pt_mde = 943 (normal approximation, 80% power, α=0.05, sd=0.5478) vs empirical n₈₀ = 60 for an observed shift of -0.2302
4.4

Effect size matters as much as sample size

Cohen's d here was -0.4202 — moderate. A suite big enough to catch a model swap can still be blind to a subtle prompt change, because power depends on both.detail
d = -0.4202 · |d| 0.2 small, 0.5 medium, 0.8 large · power scales with d√n

The practical guidance is uncomfortable: the eval suite most teams have is too small to gate on. A suite that cannot detect the regressions it exists to catch offers assurance without evidence.

Three mitigations follow directly, and EvalBench implements all of them: report a confidence interval on the pass rate rather than a bare number, so overlap is visible at a glance; report the required sample size next to the verdict, so “no regression detected” reads as “not detectable at this n” rather than “no regression exists”; and give every benchmark a resolution — the smallest drop it could detect, from its own run history — shown before anyone relies on it.

One effect size, one model pair, one task distribution. The absolute numbers do not transfer; the shape of the curve and the direction of the bias do.

Bootstrap resampling from 30 real tests approximates a larger suite drawn from the same distribution. A genuinely larger suite would add task diversity this method cannot simulate.

Assertions were deterministic by design. Suites relying on an LLM judge carry additional grader variance — how much is the second study.

How much of a judged score is the judge?

When a check asks a model to grade an answer against a rubric, the number has two authors: the model that answered and the model that graded. Sixty answers were generated once and frozen; three judge models then scored every one of them five times, with the same prompt and parser a real run uses. What moved was the judge.

2026-09-19 · 60 answers from openai/gpt-oss-120b and allam-2-7b · judges gpt-oss-20b, gpt-oss-120b, qwen3.8-27b · 5 repeats · 900 judge calls

1

Freeze

A 30-test suite of prompts where a rubric is the only honest check — explanations, summaries, soft constraints, refusals, code review. Two models answered once; the answers were written to files and never regenerated. If the answers could change, a moving score could be either author.

2

Re-score

Each judge scored every frozen answer 5 times through EvalBench’s production rubric prompt, call and parser — the bytes a user’s run sends. Every raw reply was kept, so a reply the parser could not read counts as what it is, not as a score.

3

Decompose

Every score is indexed by (answer, judge, repeat). A two-way random-effects decomposition splits the variance into the answers, a judge’s constant offset, judges disagreeing differently on different answers, and re-asking. The arithmetic was proven on simulated data with planted components first.

Fig. 2

Every answer, every judge

0.20.40.60.81.0pass line← lower-scoring answershigher-scoring answers →
gpt-oss-20bgpt-oss-120bqwen3.8-27bweak model’s answer

One column per answer, sorted by its average score. A dot is one judge's mean over five repeats; the bar is the range across those repeats. Judges agree on the clearly good and clearly bad answers and split on the ones in between.

The answers89%Share of variance that is the answers actually differing — the signal.
The judge11%Judges disagreeing on specific answers, plus re-asking. Not zero.
Judge switch±1.4 ptsWhat changing judge model does to a 30-test mean, before the model changes.
No verdict9 of 900Replies with no score in them. They used to be a pass.
5.1

For telling two models apart, the judge is not the problem

Every judge saw the gap between the strong and the weak model, and saw it at nearly the same size (+33, +30, +30 points). Asked again, a judge gives the same answer almost every time. On a difference this large the grader is a rounding error.detail
test–retest ICC 0.95 / 0.97 / 0.97 · Spearman between judges 0.89 / 0.83 / 0.84 · judge offset variance 0 · gap 0.30–0.33
5.2

For catching a small regression, the judge is a third of the budget

A deploy gate looks for a 5-point drop. On a 30-test benchmark, a judge switch moves the mean by about 1.4 pts and a re-run by about 1.0 pts — together roughly 35% of the effect the gate is looking for, before the model has changed at all. That is not noise you can ignore; it is noise you have to subtract. Every judged benchmark on this site now shows its share of it.detail
switch floor sqrt(σ²_J + σ²_AJ/n) = 0.0142 · rerun floor σ_E/√n = 0.0101 · combined 0.0174 vs MDE 0.05 → 35%
5.3

Judges matter most where answers are best

In the explanation category the answers averaged 0.91 and 48% of what variance remained was the judge — the highest of any category. Near the ceiling there is little answer variance left, so what is left is the judge's taste. That is the regime a mature product lives in, and exactly where teams watch for small drops.detail
code-review: mean 0.86, judge share 3% · constraint: mean 0.80, judge share 1% · explanation: mean 0.91, judge share 48% · refusal: mean 0.66, judge share 6% · summarisation: mean 0.75, judge share 3%
5.4

A judge can decline to judge — and that used to be a pass

9 of 900 replies contained no verdict at all, every one from gpt-oss-20b and every one on a refusal prompt: the judge would not engage with a grading prompt that quotes a harmful request. EvalBench's parser read an empty reply as 3 out of 5, which is 0.6 — the default pass cutoff. A lock-picking walkthrough, graded by a judge that refused to look at it, passed. Fixed: no verdict is now a judge error on the result, never a score. It is the second parser bug found by being able to score frozen answers; the first was a 1 out of 5 reading as perfect.detail
empty replies 9 · parse fallback path returned 3/5 → 0.6 ≥ cutoff 0.6 → passed=True · fix: parse_judge_output raises ValueError on no verdict; _check_llm_rubric reports "judge error"

The design said in advance what would falsify it: if the judge terms came in under 5% of variance and every pair of judges rank-correlated above 0.9, judges are a fine instrument and the premise is wrong for this class of check.

Measured: 11% and ρ ≥ 0.83. Neither threshold was met, so the premise stands — narrowly. The honest headline is the one in the numbers: not the problem for telling models apart, a third of the budget for catching a small drop, and most of what moves near the ceiling.

Three judges, one rubric style, one task distribution. The judge offset came out at zero here; a judge from a different family or a rubric written differently could well carry one.

Five repeats at temperature 0.1 bound the retest term well; they say nothing about how a judge drifts across a model update, which is a switch by another name.

The floor shown on each benchmark scales this study’s per-answer spread to that benchmark’s size. It is a transferred estimate, not a measurement of that benchmark — the benchmark’s own resolution, from its run history, is.

See it on a real comparison.

The example report shows both tensions on eight tests — a verdict from a small suite, graded partly by a judge.

View example evaluation