§ Example evaluation
Swapping in a cheaper model — did quality actually drop?
A real recorded run of Capability Showcase — 8 tests across factual recall, reasoning, structured output, grounded retrieval and open-ended quality — executed against two models and compared. No API key needed to read this; nothing here is illustrative.
The newer run scored 23.4 points worse, but that is within normal run-to-run variation — not a real change (p = 0.208; anything above 0.05 means it could easily be luck). To reliably detect a change this small you would need around 711 tests, not 8.
p = 0.208 — above 0.05, could easily be chance · Cohen's d -0.49 · 8 paired tests
Baseline
openai/gpt-oss-120b
Candidate
allam-2-7b
Is the difference real?
The difference is not statistically significant.detail
The effect size is small.detail
2 test(s) went pass→fail, 0 went fail→pass.detail
To resolve a move this size reliably you'd need about 711 tests.detail
The bars are 95% bootstrap confidence intervals on the pass rate. They overlap, so the true difference could plausibly be zero.
What actually moved
| Test | Baseline | Candidate | Δ |
|---|---|---|---|
| order-of-operations | 1.000 | 0.000 | -1.000 |
| phone-format | 1.000 | 0.000 | -1.000 |
| grounded-summary | 0.667 | 0.809 | +0.143 |
| explain-clearly | 0.572 | 0.560 | -0.011 |
4 test(s) scored identically.
Per-test deltas. Aggregate scores hide this — one test can collapse while another improves.
By category
| Category | Baseline | Candidate |
|---|---|---|
| factual | 100.0% | 100.0% |
| reasoning | 100.0% | 50.0% |
| structured | 100.0% | 50.0% |
| rag | 0.0% | 0.0% |
| quality | 100.0% | 100.0% |
Categories that moved: reasoning -50 pts, structured -50 pts. A single blended pass rate would hide this.
Every test from the baseline model, with each assertion that ran against it and the model’s raw response. This is the same view every run gets in the Workbench.
7 out of 8 answers passed everything you checked for; 1 missed at least one check.
95% CI 62.5%–100.0% · n=8
By category
| factual | 2/2 | 100.0% |
| reasoning | 2/2 | 100.0% |
| structured | 2/2 | 100.0% |
| rag | 1/1 | 0.0% |
| quality | 1/1 | 100.0% |
Where the model is strong or weak, by category. Tests that never got an answer are shown but not scored.
Checks run
- Matched the expected answer exactly.4✓
- Replied with correctly structured JSON.1✓
- Contains the expected answer.1✓1✗
- Matches the expected format.1✓
- Stuck to facts backed by the source material — no made-up claims.1✓
- The source material had what was needed to answer.1✓
- Means the same thing as the expected answer, even if worded differently.1✓
- Meets the grading criteria.1✓
Every kind of check used across all answers, and how many passed.
capital-japan
- ✓ Matched the expected answer exactly.
model’s actual response
Tokyo
moon-landing
- ✓ Matched the expected answer exactly.
model’s actual response
1969
arithmetic
- ✓ Matched the expected answer exactly.
model’s actual response
391
order-of-operations
- ✓ Matched the expected answer exactly.
model’s actual response
14
json-record
✓ Replied with correctly structured JSON.detail
json-schema: valid- ✓ Contains the expected answer.
model’s actual response
{
"language": "Python",
"year": 1991
}phone-format
- ✓ Matches the expected format.
model’s actual response
555-123-4567
grounded-summary
✓ Stuck to facts backed by the source material — no made-up claims.detail
faithfulness: no verifiable claims in the answer✓ The source material had what was needed to answer.detail
context-recall: 2/2 reference facts in context- ✗ Doesn't contain the expected answer.
model’s actual response
I’m sorry, but I don’t have the necessary context to answer that question.
explain-clearly
✓ Means the same thing as the expected answer, even if worded differently.detail
semantic: cosine=0.543✓ Meets the grading criteria.detail
llm-rubric: Explains the speed benefit but fails to mention any storage or write cost trade-off.
model’s actual response
An index is a data structure that stores a sorted copy of one or more columns, allowing the database to quickly locate rows matching a query without scanning the entire table. By mapping indexed values to their corresponding row pointers, it dramatically speeds up search, join, and ordering operations.
Run your own.
Pick a benchmark and a model in the Workbench — with your own key or without one.