What breaks, and the fix.
20 problems from real AI systems, each with a worked answer.
Evaluation & Observability.
Proving the new pipeline is better
Cutover to the redesigned retrieval pipeline is in two weeks. We have no labelled relevance data. How do you show it is an improvement rather than a different set of failures?
Problem 20Reproducing a failure in a non-deterministic system
A customer reported an agent failure six days ago. We have traces but cannot re-run it. Design for reproducibility.
Orchestration & Protocols.
When multi-agent is the wrong answer
A team proposes decomposing an existing single-agent workflow into seven specialized agents. Latency is currently 4 seconds and accuracy 88%. What do you ask before approving, and under what conditions do you reject it?
Problem 06Topology and context isolation
Approved: the workload genuinely needs multiple agents. Design the topology for a research task where a supervisor must dispatch 5-20 parallel searches, and explain how you stop the supervisor's context from exploding.
Problem 07Loop termination and runtime budget gates
An agent in production entered a tool-call loop and spent $4,000 in nine hours before anyone noticed. Design the controls so this cannot recur, without making the agent give up on legitimately long tasks.
Problem 08Migrating an MCP fleet to the stateless specification
We run 40 internal MCP servers on the 2025-11-25 revision. The 2026-07-28 revision removes protocol sessions and the initialize handshake entirely. Plan the migration for a fleet where several servers hold per-connection state.
Problem 09Tool namespace explosion
Our agent has access to 40 MCP servers exposing 400 tools. Tool selection accuracy has fallen below 60% and the tool definitions alone consume 60,000 tokens of every request. Fix both.
Problem 17Handoff contracts between agents
Agent A extracts requirements, hands to B which produces a plan, hands to C which executes. Post-mortems keep tracing failures to information silently lost at a handoff. Fix it.
Problem 18Compensation when an agent fails mid-flight
An agent with write access to CRM, billing and email failed after step 3 of 5. It had already issued a refund and sent a customer notification. What should the architecture have been?
Problem 19Human approval across a three-day pause
Any transaction above $10,000 requires human approval. Approvers respond in anywhere from four hours to three days. The agent currently holds an open connection and times out. Redesign.
Caching & Context Optimization.
The MCP caching contract
Under revision 2026-07-28, tools/list results carry ttlMs and cacheScope, and servers should return tools in deterministic order. Design a client-side caching layer for a fleet serving a million agent calls a day, and explain what deterministic ordering has to do with cost.
Problem 13The long-context trade point
A vendor argues we should drop retrieval entirely and use a 2 million token window. Our per-query corpus fits in about 800,000 tokens. Evaluate the proposal.
Retrieval.
Catalog retrieval at 500 million rows
Our enterprise database holds 500 million product metadata rows. Standard top-k vector retrieval is returning irrelevant chunks, which causes hallucination and blows through our token budgets. How do you re-architect this context ingestion pipeline to improve accuracy and control cost?
Problem 02Filter selectivity collapse
The same catalog search works fine for broad queries but returns near-empty or nonsensical results whenever a user applies three or more filters. Recall drops off a cliff below roughly 1% selectivity. Diagnose and fix.
Problem 03Chunking a mixed corpus
Our knowledge base is 40% PDFs with tables, 30% Confluence pages, 20% source code, 10% support tickets. One chunking configuration is used for all of it and quality is poor everywhere. What do you change?
Problem 04Freshness and incremental re-embedding
Our corpus changes 2% per day. Full re-indexing takes 14 hours and costs $8,000. Retrieval quality decays visibly between runs and finance wants the bill halved. Redesign the ingestion pipeline.
Problem 11When GraphRAG earns its indexing cost
A team wants to replace our RAG with GraphRAG after a proof of concept showed better multi-hop answers. Indexing 2 million documents with LLM-based entity extraction is quoted at $180,000 and three weeks. Approve or reject?
Problem 12The query routing layer
Every query runs the full pipeline: rewrite, hybrid retrieve, rerank, frontier model. p50 is 3.2 seconds and $0.04 per query. Inspection suggests 60% of queries are trivial and repetitive. Design the router.
Problem 14Multi-tenant isolation in a shared index
500 enterprise tenants share one vector index. Legal now requires provable isolation. The largest tenant has 40 million documents, the smallest has 200. Design it.
Problem 15Fusion is not the tuning knob
We fused BM25 and dense retrieval with reciprocal rank fusion. Aggregate metrics improved but a set of queries got noticeably worse. The team wants to tune the RRF constant. What do you tell them?