Caching & Context Optimization

Make the window cheaper and denser

You pay by the word, every single time. Caching is refusing to pay twice for the words that never change.

Most of what you send a model is the same on every request — the instructions, the company style guide, the tool descriptions. You are billed for all of it, again, on every call. Caching lets the provider keep the unchanging opening section warm so the repeat visits are far cheaper and faster. The catch is that it only works if that section is byte-for-byte identical, so a timestamp in the wrong place quietly costs you the whole discount.

15 entries · 5 categories · 4 equations

How it actually works

no jargon

Put everything that never changes at the very front: instructions, tool definitions, examples. Anything that varies — the user's actual question, today's date — goes last.

The arithmetic

Cache break-even, in reads

n^{*} \;=\; \frac{w - 1}{1 - r}

w is the write multiplier on input price, r the read multiplier. Anthropic's 5-minute cache (w = 1.25, r = 0.1) breaks even at 0.28 reads; the 1-hour cache (w = 2.0) at 1.11. Both are below one read, so caching pays from the first reuse. OpenAI charges no write premium, so n* = 0.

Effective input cost at hit rate h

C_{\text{eff}} \;=\; C_{\text{base}}\left[(1-h)\,w + h\,r\right]

At h = 0.9 with Anthropic's 5-minute cache: 0.1(1.25) + 0.9(0.1) = 0.215 — a 78.5% reduction on input spend. Hit rate, not model choice, is the largest single lever available to you.

KV cache footprint

M_{\text{KV}} \;=\; 2 \cdot L \cdot n \cdot h_{kv} \cdot d_h \cdot b \ \text{ bytes}

Two tensors (K and V) per layer L, per token n. A 70B model with L = 80, h_kv = 8, d_h = 128, fp16 holds ~327 KB per token — 21 GB at 64K context, before weights. This is the number that decides your batch size, and the reason prefix reuse dominates self-hosted economics.

Prefill vs decode cost

\underbrace{\mathcal{O}(n^2 d)}_{\text{prefill}} \qquad \underbrace{\mathcal{O}(n d)}_{\text{per decoded token}}

Prefill is quadratic in input length, decode is linear. Doubling the prompt quadruples time-to-first-token but leaves throughput roughly untouched — which is why long-context latency complaints are almost always a prefill problem that prefix caching fixes.

Provider caching

Prefix caching offered by the model vendors.

Anthropic prompt caching

Up to four cache_control breakpoints with 5-minute or 1-hour TTL, refreshed on read

Writes cost 1.25x (5-min) or 2.0x (1-hr) input. Break-even at roughly 0.28 and 1.11 reads.

anthropiccost-lever

Commercialcommercialwidely deployedsaas

Gemini explicit caching

Declared cache object with 60-minute default TTL and a large token minimum

Adds storage cost per token-hour, unlike the other two.

googlecost-lever

Commercialcommercialwidely deployedsaas

OpenAI automatic caching

Zero-config prefix cache with a 1,024-token minimum and no write surcharge

openaicost-lever

Commercialcommercialwidely deployedsaas

Self-hosted KV cache

Prefix and KV reuse in your own serving layer.

SGLang RadixAttention

Radix tree over token sequences for longest-prefix KV match, on by default

servingself-host

Apache-2.0open sourcewidely deployedself-host

vLLM prefix caching

Block-level content-hashed KV reuse, on by default in V1

servingself-host

Apache-2.0open sourcewidely deployedself-host

LMCache

Cross-instance KV cache sharing and offload to CPU or NVMe

The answer for multi-node; SGLang alone suffices single-node.

servingmulti-node

Apache-2.0open sourceproduction viableself-host

Compression & pruning

Reducing tokens before they reach the model.

Sub-agent context isolation

Spawn agents with private windows so only summaries reach the parent

A pattern rather than a product, and often more effective than a compression library.

patterncontext-isolation

Patternopen sourcewidely deployed

LLMLingua

Token-classification prompt compressor achieving 2-5x reduction

Last release April 2024, no commits since April 2026. Still recommended in roundups.

compressioncheck-before-adopting

stalledMITopen sourceresearchself-host

Token accounting

Counting and attributing spend accurately.

Anthropic count_tokens

Free endpoint returning billing-accurate counts including system prompt and tools

tokenizeranthropic

Commercial (free)commercialwidely deployedsaas

LiteLLM

Gateway normalizing token accounting and cache headers across providers

gatewaymulti-provider

MITopen sourcewidely deployedboth

tiktoken

OpenAI BPE tokenizer giving exact counts for OpenAI models only

tokenizeropenai

MITopen sourcewidely deployedself-host

Long-context evidence

What actually happens as inputs grow.

Context Rot (Chroma)

Eighteen models degrade non-uniformly as input grows; includes a replication toolkit

benchmarkcitable

Researchopen source

LongFuncEval

Long-context degradation measured specifically for tool and function calling

benchmarktool-calling

Researchopen source

Lost in the Middle

U-shaped positional accuracy, with over 30% drop for mid-context evidence

benchmarkcanonical

Researchopen source

NoLiMa

Needle-in-haystack without lexical overlap; scores collapse where standard NIAH saturates

benchmarkcitable

Researchopen source

Cache economics

live

Refreshed on every read.

cache hit rate90%

break-even reads: 0.28
effective input cost: 0.215×

78.5% off input spend

Field notes

Caching is the largest single cost lever and it is mostly free. All three major providers discount cached input reads by roughly 90%.
The discipline: put static content — system prompt, tool definitions, few-shot examples, retrieved corpus — first, and never mutate it. One changed token above a breakpoint invalidates everything below.
Anthropic charges 1.25x input to write a 5-minute cache and 2.0x for 1-hour, putting break-even at roughly 0.28 and 1.11 reads. Gemini adds storage cost per token-hour. OpenAI has no write surcharge.
Context editing and caching interact. Edits apply after cache lookup but before token counting, so clearing preserves the cached prefix but forces a new cache write. Tune trigger thresholds accordingly.
Self-hosting inverts the trade. Prefix caching is on by default in vLLM and SGLang; the real work is cache-aware routing so a session lands on the node holding its KV.
Sub-agent context isolation — spawning agents with private windows and returning only summaries — is a pattern, not a product, and is often more effective than a compression library.
Effective context is materially shorter than the advertised window. Retrieve into a focused 50-200K window and place strongest evidence at head and tail.
Cite Chroma's context rot work or NoLiMa rather than needle-in-a-haystack, which saturates.