Evals · run on every deploy
The tests behind the agent.
I tell teams to ship evals with every AI feature. These are the ones for this site’s corner agent and fit check: what they check, every case, and whether the latest run passed. The deploy stops if a score drops below its threshold.
Loading the latest run…
What is measured
Answers: the question must be answered and cite at least one expected piece of evidence. Abstains: questions the evidence cannot answer must get “no evidence”, not a guess. Fit labels: each skill in six sample job descriptions must be scored strong, partial or gap as expected. Held-out: ten questions written after the first tuning round. Later ranking fixes were checked against them too, so treat that score as optimistic. Links: every piece of evidence points at something that exists.
node tools/evals/run.mjs --live <url> scores it on the same questions, with cost per answer, and the results appear below.Ask: every question
| Result | Question | Why |
|---|
Fit brief: every label
| Result | Job description | Skill | Expected | Got |
|---|