Fusion is not the tuning knob
We fused BM25 and dense retrieval with reciprocal rank fusion. Aggregate metrics improved but a set of queries got noticeably worse. The team wants to tune the RRF constant. What do you tell them?
The constraint. RRF discards scores and uses ranks alone. That is its virtue — it needs no score normalization between two incomparable scales — and it is exactly the source of the regression. A document ranked first on overwhelming lexical evidence and one ranked first on marginal semantic similarity contribute identically.
What actually regressed. Queries where one retriever is authoritative. An exact SKU, part number or error code is a solved problem for BM25; fusion then drags in semantically plausible, factually wrong neighbours and dilutes a confident result. Tuning the constant trades one query class against another. It cannot fix this, because the information needed to fix it — how confident each retriever was — has already been thrown away.
Two real fixes.
- 01Weight per query class, using the classifier from item 12. Queries containing identifiers weight lexical heavily; natural-language queries weight dense. This is a small, interpretable change.
- 02Stop fusing for ranking at all. Use both retrievers purely as recall generators, union the candidates, and let a cross-encoder decide the order. The reranker reads the text and does not care which retriever proposed a candidate, so the entire score-comparability problem disappears. This is the cleaner architecture and it is where the accuracy is anyway.
The measurement error underneath. An aggregate nDCG improved while a query class broke, and nobody noticed until users did. Stratify every retrieval metric by query class — identifier lookup, attribute filter, natural language, multi-hop — and report per class. An aggregate over a mixed workload hides exactly the regression you need to see.
Stack. RRF ships natively in Elasticsearch, OpenSearch and Qdrant, so the fusion itself is not the work. Vespa's multi-phase ranking subsumes the whole pattern. bge-reranker-v2-m3 as the arbiter. Ranx for offline fusion comparison with proper significance testing before anything ships.
Where answers fail. Treating fusion as a hyperparameter search. The constant is not where the loss is.