05Systems and what breaks / Problems

What breaks, and the fix.

20 problems from real AI systems, each with a worked answer.

Orchestration & Protocols.

Problem 05

When multi-agent is the wrong answer

A team proposes decomposing an existing single-agent workflow into seven specialized agents. Latency is currently 4 seconds and accuracy 88%. What do you ask before approving, and under what conditions do you reject it?

Problem 06

Topology and context isolation

Approved: the workload genuinely needs multiple agents. Design the topology for a research task where a supervisor must dispatch 5-20 parallel searches, and explain how you stop the supervisor's context from exploding.

Problem 07

Loop termination and runtime budget gates

An agent in production entered a tool-call loop and spent $4,000 in nine hours before anyone noticed. Design the controls so this cannot recur, without making the agent give up on legitimately long tasks.

Problem 08

Migrating an MCP fleet to the stateless specification

We run 40 internal MCP servers on the 2025-11-25 revision. The 2026-07-28 revision removes protocol sessions and the initialize handshake entirely. Plan the migration for a fleet where several servers hold per-connection state.

Problem 09

Tool namespace explosion

Our agent has access to 40 MCP servers exposing 400 tools. Tool selection accuracy has fallen below 60% and the tool definitions alone consume 60,000 tokens of every request. Fix both.

Problem 17

Handoff contracts between agents

Agent A extracts requirements, hands to B which produces a plan, hands to C which executes. Post-mortems keep tracing failures to information silently lost at a handoff. Fix it.

Problem 18

Compensation when an agent fails mid-flight

An agent with write access to CRM, billing and email failed after step 3 of 5. It had already issued a refund and sent a customer notification. What should the architecture have been?

Problem 19

Human approval across a three-day pause

Any transaction above $10,000 requires human approval. Approvers respond in anywhere from four hours to three days. The agent currently holds an open connection and times out. Redesign.

Retrieval.

Problem 01

Catalog retrieval at 500 million rows

Our enterprise database holds 500 million product metadata rows. Standard top-k vector retrieval is returning irrelevant chunks, which causes hallucination and blows through our token budgets. How do you re-architect this context ingestion pipeline to improve accuracy and control cost?

Problem 02

Filter selectivity collapse

The same catalog search works fine for broad queries but returns near-empty or nonsensical results whenever a user applies three or more filters. Recall drops off a cliff below roughly 1% selectivity. Diagnose and fix.

Problem 03

Chunking a mixed corpus

Our knowledge base is 40% PDFs with tables, 30% Confluence pages, 20% source code, 10% support tickets. One chunking configuration is used for all of it and quality is poor everywhere. What do you change?

Problem 04

Freshness and incremental re-embedding

Our corpus changes 2% per day. Full re-indexing takes 14 hours and costs $8,000. Retrieval quality decays visibly between runs and finance wants the bill halved. Redesign the ingestion pipeline.

Problem 11

When GraphRAG earns its indexing cost

A team wants to replace our RAG with GraphRAG after a proof of concept showed better multi-hop answers. Indexing 2 million documents with LLM-based entity extraction is quoted at $180,000 and three weeks. Approve or reject?

Problem 12

The query routing layer

Every query runs the full pipeline: rewrite, hybrid retrieve, rerank, frontier model. p50 is 3.2 seconds and $0.04 per query. Inspection suggests 60% of queries are trivial and repetitive. Design the router.

Problem 14

Multi-tenant isolation in a shared index

500 enterprise tenants share one vector index. Legal now requires provable isolation. The largest tenant has 40 million documents, the smallest has 200. Design it.

Problem 15

Fusion is not the tuning knob

We fused BM25 and dense retrieval with reciprocal rank fusion. Aggregate metrics improved but a set of queries got noticeably worse. The team wants to tune the RRF constant. What do you tell them?