Retrieval architecture
Sizing, selectivity and engine choice for retrieval over a large corpus.
Our vector search returns irrelevant chunks and burns the token budget. What do we change?
Before it names a tool
Index memory budget, then the filter selectivity histogram.
The procedure
- 01
Compute the memory budget
rows × dims × bytes_per_component, plus rows × M × 8 for the graph. 500M × 1024 spans 2 TB at fp32 and 64 GB at binary — the quantization choice decides more than the engine does.
- 02
Pull the selectivity histogram
From the query log, not an estimate. Above 10% filtered ANN holds; 1–10% needs a wider search; below 1% the graph is dead ends and an exact scan is faster.
- 03
Choose the engine
Only now, and against what each one costs you — operational weight, a sync pipeline, a query language, a data model you adopt wholesale.
- 04
Fix relevance in the right stage
Fusion is recall. Precision and token spend close later: cross-encoder rerank, an absolute score threshold that can return nothing, then a hard token ceiling.
- 05
Decide RAG vs long context honestly
Long context wins on small shared corpora that cache. It loses on large per-query ones, where there is no stable prefix and every call pays the cache-write multiplier.
- 06
Prove it before cutting over
recall@k is the ceiling on everything downstream. Bootstrap labels with inverse cloze, then shadow-and-diff on the queries where old and new disagree.
Where answers fail
- Naming an engine before computing the memory budget.
- Treating reciprocal rank fusion as a precision fix. It is recall.
- Estimating selectivity instead of reading it off the query log.
- Reporting generation quality as evidence that retrieval improved.
- Assuming long context is cheaper because the corpus fits in the window.
Install
One markdown file. Drop it in your agent’s skills directory and it loads when the description matches what you are doing.
mkdir -p ~/.claude/skills/retrieval-architecture
curl -o ~/.claude/skills/retrieval-architecture/SKILL.md \
https://tokalator.wiki/applied-ai/skills/retrieval-architecture.md