The query routing layer
Every query runs the full pipeline: rewrite, hybrid retrieve, rerank, frontier model. p50 is 3.2 seconds and $0.04 per query. Inspection suggests 60% of queries are trivial and repetitive. Design the router.
The constraint. A router is a classifier, and a classifier that costs as much as the path it is protecting has negative value. The router must be at least two orders of magnitude cheaper than the expensive path or the arithmetic never closes. That rules out using a frontier model to route.
Four tiers, cheapest first.
| Tier | Mechanism | Cost | Typical share |
|---|---|---|---|
| Exact cache | Normalize, hash, look up | ~0 | 10-20% |
| Semantic cache | Embed, threshold on nearest neighbour | one embedding call | 10-25% |
| Rules | Regex and keyword for known intents | ~0 | more than teams expect |
| Small classifier | Fine-tuned encoder, ~150M params, 5-10 ms on CPU | negligible | the remainder |
Only what survives all four reaches the frontier model.
The semantic cache is where this goes wrong. Embeddings that are close are not the same intent. "How do I cancel my subscription" and "how do I cancel a cancellation" sit within a few hundredths of cosine distance and have opposite answers. Three controls: a threshold set from a labelled set rather than by intuition, never caching a personalised or stateful answer, and a per-entry TTL tied to how fast the underlying content changes.
Escalation must exist and must be measured. The cheap path needs a way to say "not confident" and hand up. Without it, routing converts a latency win into a silent quality regression. The metric that matters is not router accuracy — it is the false-cheap rate, the fraction of queries sent down the cheap path that should have gone up. Weight it heavily, because a query wrongly sent to the expensive path costs four cents while one wrongly sent to the cheap path costs a customer.
Stack. Redis or GPTCache for the semantic cache tier. A fine-tuned ModernBERT or DeBERTa encoder for the classifier — this is a classification problem with abundant training data sitting in your logs, not a generation problem. LiteLLM or RouteLLM for the routing layer itself. Langfuse to attribute quality regressions back to routing decisions, which requires recording the tier on every trace.
Where answers fail. Proposing an LLM as the router. It reintroduces most of the cost and all of the latency variance you were trying to remove.