Proving the new pipeline is better
Cutover to the redesigned retrieval pipeline is in two weeks. We have no labelled relevance data. How do you show it is an improvement rather than a different set of failures?
The constraint. End-to-end answer quality is the wrong first metric. It conflates retrieval failure with generation failure, and when it drops you cannot tell which half to fix. Decompose before measuring.
Three metrics, in dependency order.
| Stage | Metric | Why it comes first |
|---|---|---|
| Retrieval | recall@k, where k is what you actually feed the model | A hard ceiling on everything downstream |
| Reranking | nDCG@10, MRR | Whether the right document reaches the window |
| Generation | Groundedness given the retrieved set | Only meaningful once the set is right |
If recall@20 is 60%, no prompt work reaches 61%. Most teams that believe they have a hallucination problem have a recall problem, and this ordering surfaces that in an afternoon.
Getting labels in two weeks without annotators.
- 01Mine the logs. Clicks, purchases, ticket resolutions and copy events are implicit relevance judgements, and you already have millions. Noisy, free, and large enough for recall measurement.
- 02Generate questions from documents. Inverse cloze: take a passage, generate a question it answers, and the source passage is the label. Produces a large set cheaply. Biased toward extractive questions, so it supplements a real set rather than replacing one.
- 03Hand-label 200 stratified examples drawn from real queries, spread across query classes. One day of work, and it is the set that catches regressions the other two miss.
The cutover mechanism. Shadow the new pipeline on live traffic, retrieve with both, log both candidate sets, ship neither. Where the sets agree, no risk. Where they disagree, sample and review — that population is small and it is the only part that carries information. This produces a defensible decision in days without a labelled corpus, and it doubles as the regression harness afterwards.
Stack. Ranx or trec_eval for IR metrics with significance testing. RAGAS for the generation-stage measures. promptfoo for the comparison matrix in CI. Langfuse for the shadow traces and the disagreement sampling.
Where answers fail. Reaching for an LLM judge before measuring recall. The judge will confirm the answers are bad without telling you that the right document was never retrieved.