Inside I · Context engineering
The full index
Every entry across the six layers in one place. Narrow by topic, layer, licence model, maturity, deployment or project health — topics stack, so picking two shows only the entries carrying both. Maturity and status are my readings of the evidence; every row links to the source.
AGENTS.md
Tells coding agents how to build, test and style a repository
Read by Codex, Cursor, Copilot, Gemini CLI, Zed, Aider and goose. No ratified versioned spec yet.
Anthropic count_tokens
Free endpoint returning billing-accurate counts including system prompt and tools
Anthropic prompt caching
Up to four cache_control breakpoints with 5-minute or 1-hour TTL, refreshed on read
Writes cost 1.25x (5-min) or 2.0x (1-hr) input. Break-even at roughly 0.28 and 1.11 reads.
Arize Phoenix
OTel-native tracing and eval workflow that runs fully local
Azure AI Search
Hybrid BM25, vector and semantic reranking with integrated vectorization
Bedrock Knowledge Bases
Managed ingestion and retrieval over OpenSearch, Aurora or Neptune
BGE (BAAI)
Open-weight embeddings; bge-reranker-v2-m3 is the common self-host default
Braintrust
Eval-first workflow with scoring, prompt playground and a VPC data plane
Proprietary schema first-class; OTel is an export path, not the internal model.
CLAUDE.md
Hierarchical project context with imports and per-directory scoping
Cohere Embed v4 / Rerank 4
Multimodal embeddings with Matryoshka dims, plus a cross-encoder rerank API
Datadog LLM Observability
LLM spans inside existing APM, correlated with infrastructure telemetry
Usually already on the approved-vendor list, which matters more than features.
Elasticsearch
BM25 plus dense and sparse retrieval, RRF hybrid, retrievers API
Added AGPL-3.0 in 2024, restoring OSI-approved status.
FAISS
ANN index library rather than a database; baseline for offline search
Gemini explicit caching
Declared cache object with 60-minute default TTL and a large token minimum
Adds storage cost per token-hour, unlike the other two.
Instructor
Pydantic validation plus a retry loop around any provider
Langfuse
Tracing, prompt management, evals and datasets with the strongest self-host story
Emits OTel-native spans. Survives enterprise procurement.
LangGraph
Graph runtime with checkpointed state machines, durable execution and human-in-the-loop
LangSmith
Trace and debug LangChain and LangGraph agents, with datasets and online evals
Self-hosting is gated behind the Enterprise tier.
LiteLLM
Gateway normalizing token accounting and cache headers across providers
LlamaParse
VLM-based PDF parsing to Markdown; best-in-class on messy documents
MCP
Client/server protocol for exposing tools, resources and prompts to models
Donated by Anthropic to the Linux Foundation in December 2025, removing single-vendor risk. Tool poisoning remains an OWASP-catalogued attack class.
Microsoft Presidio
NER and regex PII detection, redaction and anonymisation
Deterministic, which makes it one of the few components here that behaves predictably.
Milvus / Zilliz
Distributed billion-scale vector database; an LF AI & Data project
MongoDB Atlas Vector Search
Vector index inside MongoDB with native embedding and rerank APIs
OpenAI automatic caching
Zero-config prefix cache with a 1,024-token minimum and no write surcharge
OpenSearch
Apache-2.0 fork with k-NN, RRF hybrid and neural sparse retrieval
pgvector
Postgres extension: HNSW/IVFFlat vector indexes in your existing database
Pinecone
Serverless managed vector database with hybrid search and namespaces
Qdrant
Rust vector database with payload filtering, quantization and on-disk HNSW
RAGAS
RAG-component metrics: faithfulness, context precision and recall, answer relevancy
Scores a single retrieve-then-generate turn. No trajectory primitives.
SGLang RadixAttention
Radix tree over token sequences for longest-prefix KV match, on by default
Sub-agent context isolation
Spawn agents with private windows so only summaries reach the parent
A pattern rather than a product, and often more effective than a compression library.
Tantivy
Rust full-text search library embedded in many higher-level tools
Temporal
General durable execution treating agents as replayable workflows
Infrastructure rather than an agent library. Often the right substrate underneath one.
tiktoken
OpenAI BPE tokenizer giving exact counts for OpenAI models only
Unstructured
Thirty-plus formats to normalized elements, with connectors and VPC deployment
Vertex AI Search
Managed retrieval and grounding with connectors and a ranking API
Vespa
Search and ranking engine with tensor compute and ML ranking phases
vLLM prefix caching
Block-level content-hashed KV reuse, on by default in V1
Weaviate
Native BM25 and dense hybrid search, multi-tenancy, module system
XGrammar
Pushdown-automaton constrained decoding, the default in vLLM, SGLang and TRT-LLM
A2A
Peer agent discovery via Agent Cards, plus task delegation
Anthropic context editing
Server-side clearing of stale tool results and thinking blocks before token counting
Anthropic memory tool
Claude issues CRUD against a /memories directory; you own the storage backend
Stores nothing itself. Budget for path-traversal defenses, per-tenant isolation and retention policy.
Azure Prompt Shields
Managed moderation and prompt-injection filters
Published work reports character-injection evasion approaching total bypass.
Chonkie
Chunking library covering token, semantic, recursive and late chunking
Chroma
Embedded single-node vector store with a Rust core
Claude Agent SDK
The harness behind Claude Code: subagents, hooks, compaction, MCP-native
ColBERT / PLAID
Late-interaction retrieval with token-level MaxSim; higher index cost
Comet Opik
Tracing, evaluation and guardrail hooks with no enterprise feature gating
Credo AI
Risk registers, control mappings and Annex IV technical documentation
DeepEval
Pytest-style LLM unit tests covering RAG, conversational and agent metrics
Docling
IBM Research layout and table pipeline; local PDF-to-Markdown, no per-page cost
Firecrawl
Crawls sites to clean Markdown or JSON and handles JS rendering
AGPL — check policy before embedding.
FlashRank
Small CPU-only reranker for latency-constrained pipelines
Gemini context caching
Explicit cached-content handles or implicit auto-cache on repeated prefixes
Google ADK
Agent SDK with evaluation and a deployment path into Vertex Agent Engine
Graphiti
Temporal knowledge graph with edge validity intervals and hybrid graph-vector search
Graphiti
Bi-temporal graph where edges carry valid-from and invalid-at, so facts expire
Temporal invalidation is the clearest differentiator against native memory tools.
Guardrails AI
Output validators and schema enforcement via a validator hub
Helicone
Proxy-based logging and caching with cache-hit-rate dashboards
Acquired by Mintlify in March 2026 and now in maintenance mode. Plan migration if you are on it.
Holistic AI
AI risk assessment, bias auditing and governance reporting
HoneyHive
Tracing, evaluation and dataset curation with VPC deployment
IBM watsonx.governance
Model inventory, risk workflows and compliance evidence inside the IBM stack
Inspect AI
UK AI Security Institute harness with solvers and scorers, strong for safety evals
Jina AI
Open-weight embeddings and multimodal rerankers that score page images
Lakera Guard
Commercial injection and jailbreak detection API
LanceDB
Serverless vector database on the Lance columnar format, reads from object storage
LangChain 1.x
Integration layer, now a thin facade over the LangGraph runtime
Third significant rewrite. Adopt LangGraph directly and treat integrations as optional glue.
Llama Guard / Prompt Guard
Hazard-taxonomy classifier plus a fast injection pre-filter
Defeated by prompt-overflow inputs that it catches in short context.
LlamaIndex
Ingestion, parsing, indexing and retrieval, with agent features added later
A retrieval library with agent features, not an orchestration spine.
llguidance
Low-latency grammar engine with the strongest measured JSON validity
LLM Guard
Scanner suite covering PII, toxicity and injection heuristics
LMCache
Cross-instance KV cache sharing and offload to CPU or NVMe
The answer for multi-node; SGLang alone suffices single-node.
Marker
PDF and EPUB to Markdown, GPU-accelerated and batch-oriented
Mastra
TypeScript-native agents, workflows, RAG and evals
Meilisearch
Lightweight full-text and hybrid search for in-application search
Mem0
Extracts facts into a vector, graph and KV store scoped by user, session or agent
Microsoft Agent Framework
AutoGen and Semantic Kernel merged; .NET and Python agents plus workflow graphs
Microsoft GraphRAG
Entity extraction, community detection and hierarchical summaries
Indexing cost is substantial at corpus scale.
Mixedbread
Open-weight embedding and reranker models plus a hosted API
MLflow Tracing
GenAI tracing inside an existing MLflow registry estate
NeMo Guardrails
Colang dialog-flow rails and topic control that orchestrate other classifiers
neo4j-graphrag-python
Official Neo4j retrievers and knowledge-graph construction pipeline
OpenAI Agents SDK
Minimal agent loop with handoffs, guardrails, tracing and sessions
OpenAI Evals
Registry-based eval templates, OpenAI-centric
OpenAI Responses API state
Server-stored response items chained via previous_response_id, 30-day retention
Outlines
FSM-based structured generation, largely displaced by XGrammar
ParadeDB
Postgres extension adding Tantivy-backed BM25 for in-database hybrid search
AGPL — check policy before embedding.
pgvectorscale
StreamingDiskANN index layered on pgvector for larger-than-memory workloads
promptfoo
Declarative YAML eval matrices plus automated red-teaming
Pydantic AI
Type-safe agent layer with schema validation and dependency injection first-class
Qwen3-Embedding / Reranker
Open-weight embedding and reranker models topping MTEB in 2026
Reducto
Agentic OCR-correction parsing with on-prem, SOC 2 Type II and HIPAA
Turbopuffer
Object-storage-native search with SPFresh index, BM25 and sparse vectors
Typesense
Typo-tolerant keyword search with vector fields and a simple ops model
Vectara
Managed RAG with HHEM hallucination scoring; enterprise-only since 2026
Self-service tier discontinued in 2026.
VectorChord
Postgres extension with disk-friendly IVF/RaBitQ; successor to pgvecto.rs
AGPL — check policy before embedding.
Voyage AI
Domain-tuned embeddings and rerankers for legal, code and finance
Acquired by MongoDB, February 2025.
W&B Weave
Trace and eval layer atop Weights & Biases
Zep
Async summarization and entity extraction layered over Graphiti
AGNTCY
Directory, identity, messaging and observability for agent meshes
Ax
TypeScript equivalent of DSPy's signature-driven programs
Chunkr
Parsing and semantic chunking API, self-hostable
AGPL — check policy before embedding.
Cognee
Pipeline turning documents into a queryable memory graph
Cognee
ECL pipeline building an ontology-typed graph alongside a vector index
DBOS
Durable execution using Postgres as the workflow source of truth
GEPA
Reflective evolutionary prompt optimization, shipping inside DSPy
Hindsight
Retain, recall and reflect over memory banks with fact extraction and an MCP server
LangMem
Extracts and updates memories into a LangGraph store
Recent activity is largely dependency bumps, with no tagged releases.
Letta (ex-MemGPT)
LLM-as-OS paging between main context, recall and archival stores
Core repo took ~5 commits in three months; development moved to sibling repos. Still widely recommended in roundups.
LightRAG
Flat dual-level graph index, far cheaper to build than GraphRAG
llms.txt
Site-level markdown index intended for LLM consumers
No major AI platform has confirmed reading it. Optional SEO hygiene, not context engineering.
MemOS
OS-style abstraction over plaintext, activation (KV) and parametric memory
OTel GenAI conventions
Semantic conventions for gen_ai.* spans, metrics and events
Every document in the spec repo is still marked Development as of July 2026. Emit OTel anyway as a lock-in hedge.
Ragie
Managed ingestion and retrieval API with connectors and citations
Supermemory
Memory API and RAG over user context, with a router SDK and MCP
Trustible
Regulatory mapping and AI use-case registry for compliance teams
ZeroEntropy
zerank reranker models at the top of several 2026 leaderboards
AutoGen
Research multi-agent conversation framework, superseded by Agent Framework
Maintenance mode. Greenfield work here is building on a dead branch.
DSPy
Declarative signatures and modules that compile prompts against a metric
Offline optimizers, Python-only, no gateway or observability story.
LLMLingua
Token-classification prompt compressor achieving 2-5x reduction
Last release April 2024, no commits since April 2026. Still recommended in roundups.
Semantic Kernel
.NET-first agent and plugin SDK, folded into Agent Framework
Maintenance mode since Agent Framework GA in April 2026.
TextGrad
Backpropagates natural-language gradients through LLM pipelines
A paper you can import.
CrewAI
Role-based multi-agent crews with LLM-driven task routing
LLM-driven routing makes cost and latency non-deterministic and failures hard to reproduce. Widely downloaded, widely abandoned at the pilot boundary.
Context Rot (Chroma)
Eighteen models degrade non-uniformly as input grows; includes a replication toolkit
Digital Omnibus on AI
First AI Act amendment; Regulation (EU) 2026/1744, in force 27 July 2026
Deferred Annex III high-risk obligations to 2 Dec 2027 and Annex I to 2 Aug 2028. GPAI enforcement on 2 Aug 2026 was NOT deferred.
EU AI Act
Risk-tiered obligations with extraterritorial reach; Regulation (EU) 2024/1689
In force 1 Aug 2024. Prohibitions and AI literacy live 2 Feb 2025. GPAI obligations live 2 Aug 2025.
ISO/IEC 42001:2023
Certifiable AI management system standard
The only auditable certificate in this space. Takes 6-12 months and increasingly appears in due-diligence questionnaires.
LongFuncEval
Long-context degradation measured specifically for tool and function calling
Lost in the Middle
U-shaped positional accuracy, with over 30% drop for mid-context evidence
NIST AI RMF 1.0
Voluntary US risk framework with a Generative AI profile
The cheapest credible starting point, and it maps onto ISO 42001. Roughly 3-6 months.
NoLiMa
Needle-in-haystack without lexical overlap; scores collapse where standard NIAH saturates