13Caching & Context Optimization
The long-context trade point
A vendor argues we should drop retrieval entirely and use a 2 million token window. Our per-query corpus fits in about 800,000 tokens. Evaluate the proposal.
The constraint. Effective context is materially shorter than advertised context, and this is measured rather than folklore.
| Evidence | Finding |
|---|---|
| NoLiMa (arXiv 2502.05167) | Performance falls sharply past 32K even in models advertising 128K and beyond, once lexical overlap between question and answer is removed |
| Lost in the Middle (arXiv 2307.03172) | Accuracy degrades for evidence placed mid-context and recovers at the head and tail |
| LongFuncEval (arXiv 2505.10570) | Tool-calling degrades faster with context length than prose question answering — directly relevant to agents |
| Chroma Context Rot | Degradation is continuous with length, not a cliff at the documented limit |
The arithmetic. 800,000 tokens per query against 5,000 retrieved tokens is a 160x cost difference. Prompt caching does not rescue it: the cache discount applies to a stable prefix, and a per-query corpus is by definition not stable. You would be paying the cache write multiplier on every query, which is worse than not caching. Prefill is also roughly linear in tokens, so 800K puts time-to-first-token into whole seconds before the model has produced anything.
Where long context genuinely wins. A single document that cannot be chunked without destroying it — a contract, a long statute, one large source file. A stable prefix reused across many queries, where caching does apply. Low query volume where engineering time costs more than tokens. And cases where retrieval recall is the bottleneck and you would rather over-retrieve than tune.
The synthesis, which is the answer. Long context does not replace retrieval; it makes retrieval cheaper to get right. A larger window tolerates a larger k, less aggressive chunking, and fewer precision heroics. Retrieve into a focused 50,000 to 200,000 token window and place the strongest evidence at head and tail. That captures nearly all of the accuracy benefit at a fraction of the cost, and it degrades gracefully when the corpus grows past any window.
Where answers fail. Accepting or rejecting on principle. Ask for the vendor's evidence at 800K on your task, then reproduce it with the needle placed at 60% depth rather than at the end.