Design problems

The constraint that decides it

Each problem is stated with numbers, because the numbers are what separate a real answer from a diagram. Every one names the constraint that settles it before it names a tool, and ends with where answers fail.

20 written · 50 planned

01Retrieval

01Catalog retrieval at 500 million rowsOur enterprise database holds 500 million product metadata rows. Standard top-k vector retrieval is returning irrelevant chunks, which causes hallucination and blows through our token budgets. How do you re-architect this context ingestion pipeline to improve accuracy and control cost?
02Filter selectivity collapseThe same catalog search works fine for broad queries but returns near-empty or nonsensical results whenever a user applies three or more filters. Recall drops off a cliff below roughly 1% selectivity. Diagnose and fix.
03Chunking a mixed corpusOur knowledge base is 40% PDFs with tables, 30% Confluence pages, 20% source code, 10% support tickets. One chunking configuration is used for all of it and quality is poor everywhere. What do you change?
04Freshness and incremental re-embeddingOur corpus changes 2% per day. Full re-indexing takes 14 hours and costs $8,000. Retrieval quality decays visibly between runs and finance wants the bill halved. Redesign the ingestion pipeline.
11When GraphRAG earns its indexing costA team wants to replace our RAG with GraphRAG after a proof of concept showed better multi-hop answers. Indexing 2 million documents with LLM-based entity extraction is quoted at $180,000 and three weeks. Approve or reject?
12The query routing layerEvery query runs the full pipeline: rewrite, hybrid retrieve, rerank, frontier model. p50 is 3.2 seconds and $0.04 per query. Inspection suggests 60% of queries are trivial and repetitive. Design the router.
14Multi-tenant isolation in a shared index500 enterprise tenants share one vector index. Legal now requires provable isolation. The largest tenant has 40 million documents, the smallest has 200. Design it.
15Fusion is not the tuning knobWe fused BM25 and dense retrieval with reciprocal rank fusion. Aggregate metrics improved but a set of queries got noticeably worse. The team wants to tune the RRF constant. What do you tell them?

02Memory & State

No worked problem yet. What should the system still know tomorrow?

03Caching & Context Optimization

04Orchestration & Protocols

05Evaluation & Observability

06Guardrails & Governance

No worked problem yet. What stops this from causing harm, and who signs off?

Still to write

21-26Protocol. server/discover fleet negotiation, MRTR, tasks extension, supply chain, authorization, transport failure.
27-33Cost. Cache-aware prompts, model cascades, budget enforcement, batch vs realtime, fine-tune break-even, self-host break-even, chargeback.
34-39Latency and serving. p99 decomposition, streaming, backpressure, KV-aware routing, capacity planning, cold start.
40-43Memory. Session vs long-term memory, temporal invalidation, erasure, state explosion.
44-46Evaluation. Bootstrapping evals, trajectory scoring, judge calibration.
47-50Governance. Injection through retrieved content, tenant PII and residency, regulatory classification, incident response.