Evals · run on every deploy

The tests behind the agent.

I tell teams to ship evals with every AI feature. These are the ones for this site’s corner agent and fit check: what they check, every case, and whether the latest run passed. The deploy stops if a score drops below its threshold.

Loading the latest run…

What is measured

Answers: the question must be answered and cite at least one expected piece of evidence. Abstains: questions the evidence cannot answer must get “no evidence”, not a guess. Fit labels: each skill in six sample job descriptions must be scored strong, partial or gap as expected. Held-out: ten questions written after the first tuning round. Later ranking fixes were checked against them too, so treat that score as optimistic. Links: every piece of evidence points at something that exists.

What it does not measure yet: how well Claude writes, since that needs the live endpoint. When it is switched on, node tools/evals/run.mjs --live <url> scores it on the same questions, with cost per answer, and the results appear below.

Ask: every question

ResultQuestionWhy

Fit brief: every label

ResultJob descriptionSkillExpectedGot