Skip to content
awesome-applied-ai

Research

The arguments, and what settles them

Most applied AI disagreements are arithmetic wearing an opinion. This page collects the findings that contradict the received wisdom, the six layers every request passes through, and the 20 equations that decide the arguments one at a time.

6 findings · 6 layers · 20 equations

Where the consensus is wrong

6 findings

The stack

Stages 02 through 07 of the cycle all pass through the same six layers, ordered the way a request flows: evidence in at the base, constraint and proof at the top.

06

Guardrails & Governance

Constrain it and prove it

Guardrails & PII7Governance tooling4Regulation & frameworks4
15
05

Evaluation & Observability

Know whether any of it works

Tracing11Evaluation frameworks5
16
04

Orchestration & Protocols

Assemble context, coordinate tools

Agent frameworks14Prompt optimization4Protocols & conventions6Structured output4
28
03

Caching & Context Optimization

Make the window cheaper and denser

Provider caching3Self-hosted KV cache3Compression & pruning2Token accounting3Long-context evidence4
15
02

Memory & State

Decide what survives the turn

Memory systems9Native platform state4
13
01

Retrieval

Find what belongs in the window

Vector databases13Search & hybrid retrieval6Embeddings & rerankers9Parsing & ingestion8Managed retrieval5Graph retrieval5
46

The arithmetic

20 equations

01Retrieval

  • Reciprocal rank fusion

    RRF(d)  =  ∑r∈R1k+r(d),k=60\mathrm{RRF}(d) \;=\; \sum_{r \in R} \frac{1}{k + r(d)}, \qquad k = 60

    How Elasticsearch and OpenSearch fuse BM25 with dense results. It uses ranks, not scores, so you never have to normalise incomparable similarity scales. k = 60 comes from Cormack et al. 2009 and damps top-rank dominance — a document ranked 1st lexically and 50th semantically still surfaces.

  • Late interaction — MaxSim

    Sq,d  =  ∑i∈∣q∣max⁡j∈∣d∣  Eqi⋅Edj⊤S_{q,d} \;=\; \sum_{i \in |q|} \max_{j \in |d|} \; E_{q_i} \cdot E_{d_j}^{\top}

    ColBERT scores every query token against its best-matching document token instead of collapsing both to one vector. Recall improves; the index stores a vector per token, so it runs one to two orders of magnitude larger. That storage bill is the whole trade.

  • HNSW resident memory

    M  ≈  N (4d+8m)M \;\approx\; N\,(4d + 8m)

    N vectors of d float32 dimensions plus roughly m graph links per node. Ten million 1536-dim vectors at m = 16 need ~63 GB of RAM. That figure — not query latency — is what pushes teams to quantization or to object-storage-native engines.

  • Recall / latency knob

    recall↗ saturating,latency∼O(efSearch⋅log⁡N)\text{recall} \nearrow \text{ saturating},\quad \text{latency} \sim \mathcal{O}(\mathrm{efSearch}\cdot \log N)

    Recall saturates in efSearch while latency keeps climbing roughly linearly. Measure the knee on your own data; the defaults shipped by every vector database are tuned for benchmark recall, not your p99.

02Memory & State

  • Bi-temporal fact

    f  =  (s, p, o,  [ tvalid, tinvalid ),  tingest)f \;=\; \big(s,\, p,\, o,\; [\,t_{\text{valid}},\, t_{\text{invalid}}\,),\; t_{\text{ingest}}\big)

    Two independent time axes: when the fact was true in the world, and when the system learned it. Drop the second and you cannot reconstruct what the agent believed at the moment it acted — which is exactly what an incident review asks for. This interval is Graphiti's actual differentiator over flat fact stores.

  • Recency-weighted recall

    score(m)  =  λ sim(q,m)  +  (1−λ) e−Δt/τ\mathrm{score}(m) \;=\; \lambda\,\mathrm{sim}(q, m) \;+\; (1-\lambda)\,e^{-\Delta t / \tau}

    Almost every memory layer implements some version of this. The failure mode is structural: a stale fact with high similarity outranks a fresh correction, because exponential decay only discounts age — it never marks the old fact false. Temporal invalidation does.

  • Memory selection is a knapsack

    max⁡∑ivixis.t.∑icixi≤B,    xi∈{0,1}\max \sum_i v_i x_i \quad \text{s.t.} \quad \sum_i c_i x_i \le B,\;\; x_i \in \{0,1\}

    B is the token budget you allocate to memory, not the model's advertised window. Every memory system is solving this approximately and none of them will tell you what B is — set it yourself.

03Caching & Context Optimization

  • Cache break-even, in reads

    n∗  =  w−11−rn^{*} \;=\; \frac{w - 1}{1 - r}

    w is the write multiplier on input price, r the read multiplier. Anthropic's 5-minute cache (w = 1.25, r = 0.1) breaks even at 0.28 reads; the 1-hour cache (w = 2.0) at 1.11. Both are below one read, so caching pays from the first reuse. OpenAI charges no write premium, so n* = 0.

  • Effective input cost at hit rate h

    Ceff  =  Cbase[(1−h) w+h r]C_{\text{eff}} \;=\; C_{\text{base}}\left[(1-h)\,w + h\,r\right]

    At h = 0.9 with Anthropic's 5-minute cache: 0.1(1.25) + 0.9(0.1) = 0.215 — a 78.5% reduction on input spend. Hit rate, not model choice, is the largest single lever available to you.

  • KV cache footprint

    MKV  =  2⋅L⋅n⋅hkv⋅dh⋅b  bytesM_{\text{KV}} \;=\; 2 \cdot L \cdot n \cdot h_{kv} \cdot d_h \cdot b \ \text{ bytes}

    Two tensors (K and V) per layer L, per token n. A 70B model with L = 80, h_kv = 8, d_h = 128, fp16 holds ~327 KB per token — 21 GB at 64K context, before weights. This is the number that decides your batch size, and the reason prefix reuse dominates self-hosted economics.

  • Prefill vs decode cost

    O(n2d)⏟prefillO(nd)⏟per decoded token\underbrace{\mathcal{O}(n^2 d)}_{\text{prefill}} \qquad \underbrace{\mathcal{O}(n d)}_{\text{per decoded token}}

    Prefill is quadratic in input length, decode is linear. Doubling the prompt quadruples time-to-first-token but leaves throughput roughly untouched — which is why long-context latency complaints are almost always a prefill problem that prefix caching fixes.

04Orchestration & Protocols

  • Reliability compounds multiplicatively

    Psuccess  =  ∏i=1npi  =  p nP_{\text{success}} \;=\; \prod_{i=1}^{n} p_i \;=\; p^{\,n}

    A 20-step agent whose every step is 99% reliable succeeds 81.8% of the time. At 95% per step it succeeds 35.8%. This single line is the entire argument for durable execution — you cannot reach acceptable end-to-end reliability by improving prompts.

  • Expected steps executed, with and without checkpoints

    E[S]retry-all=np nE[S]checkpoint=npE[S]_{\text{retry-all}} = \frac{n}{p^{\,n}} \qquad E[S]_{\text{checkpoint}} = \frac{n}{p}

    At n = 20 and p = 0.95, restarting the whole run costs ~56 step-executions; resuming from the last checkpoint costs ~21. The gap widens exponentially in n, which is why the durability substrate is the decision and the agent library is the replaceable part.

  • Grammar-constrained decoding

    p′(t)  =  p(t) 1 ⁣[t∈Vvalid]∑t′∈Vvalidp(t′)p'(t) \;=\; \frac{p(t)\,\mathbb{1}\!\left[t \in V_{\text{valid}}\right]}{\displaystyle\sum_{t' \in V_{\text{valid}}} p(t')}

    Mask invalid tokens, renormalise. Schema conformance becomes 100% by construction — which is why JSON validity is a solved non-issue. The mask says nothing about whether the model picked the right tool or plausible-but-wrong arguments.

05Evaluation & Observability

  • Judge agreement — Cohen's κ

    κ  =  po−pe1−pe\kappa \;=\; \frac{p_o - p_e}{1 - p_e}

    Observed agreement with human labels, corrected for chance. On a balanced binary task a judge that agrees 80% of the time scores κ ≈ 0.6; at 60% it scores κ ≈ 0.2 and is close to noise. Never ship an LLM judge without this number.

  • Position-swap consistency

    Cswap  =  Pr⁡[ j(a,b)=j(b,a) ]C_{\text{swap}} \;=\; \Pr\big[\, j(a,b) = j(b,a) \,\big]

    Run every pairwise comparison in both orders. High C_swap is what teams mistake for accuracy — a judge can be perfectly self-consistent and consistently wrong. κ measures correctness, C_swap only measures stability.

  • Faithfulness

    faithfulness  =  ∣{ c∈C  :  R⊨c }∣∣C∣\text{faithfulness} \;=\; \frac{\big|\{\, c \in C \;:\; R \models c \,\}\big|}{|C|}

    Fraction of claims C in the answer entailed by retrieved context R. Scores one retrieve-then-generate turn, which is why RAG metrics are commoditised and agent trajectory scoring is not.

  • Trajectory vs outcome

    Pr⁡[path correct]  ≤  Pr⁡[answer correct]\Pr[\text{path correct}] \;\le\; \Pr[\text{answer correct}]

    Outcome accuracy is an upper bound on trajectory accuracy, never a proxy for it. The gap is where broken tool calls hide behind lucky answers.

06Guardrails & Governance

  • Layered defense, attack success rate

    ASRtotal  =  ∏i=1kASRionly if ASRi⊥ASRj\mathrm{ASR}_{\text{total}} \;=\; \prod_{i=1}^{k} \mathrm{ASR}_i \quad \text{only if } \mathrm{ASR}_i \perp \mathrm{ASR}_j

    Three filters at 30% ASR each multiply to 2.7% — on paper. Independence is the assumption that fails: an adaptive attacker who defeats one classifier usually defeats correlated ones, which is why measured layered ASR against adaptive attacks stays above 85% instead of collapsing.

  • False blocks per day

    Bfalse  =  FPR×QdayB_{\text{false}} \;=\; \mathrm{FPR} \times Q_{\text{day}}

    A filter with 1% FPR on a million-request/day product blocks 10,000 legitimate requests. Guardrail vendors quote recall; the number that decides whether you can ship is FPR at your traffic volume.

The map

1 / 5 written
I

Context engineering

written

What goes into the window, what it costs to put it there, and what you can prove about it afterwards.

  • retrieval
  • memory & state
  • caching
  • orchestration
  • evaluation
  • guardrails
  • Guardrails & Governance15
  • Evaluation & Observability16
  • Orchestration & Protocols28
  • Caching & Context Optimization15
  • Memory & State13
  • Retrieval46
133 entries · 6 layers
II

Enterprise problems & solutions

drafting

The problem as the business states it, the constraint that actually binds, and the thing that breaks six weeks after launch.

  • problem catalogue
  • reference architectures
  • failure post-mortems
  • build vs buy
  • procurement traps
0 of ~40 planned
III

Model & serving economics

planned

Which model, on whose hardware, at what tail latency, and what a thousand requests actually cost.

  • routing & fallback
  • quantisation
  • batching & throughput
  • fine-tune vs prompt
  • unit economics
0 of ~45 planned
IV

Data foundations

planned

What you are retrieving from, whether it is clean enough to retrieve from, and whether you are allowed to.

  • ingestion & parsing
  • chunking
  • labelling
  • synthetic data
  • lineage & residency
0 of ~50 planned
V

Delivery & operating model

planned

Who ships it, who gets paged at 3am, and who signs the risk acceptance before it reaches a customer.

  • team shapes
  • rollout & canarying
  • incident response
  • model risk sign-off
  • vendor management
0 of ~30 planned
133
indexed entries
  • Guardrails15
  • Evaluation16
  • Orchestration28
  • Caching15
  • Memory13
  • Retrieval46
6 layers
71%
open or hybrid licence
  • open source69
  • open core25
  • source available4
  • commercial35
35 closed
81%
production-grade
  • widely deployed41
  • production viable60
  • early18
  • research5
  • fragile1
1 fragile
6
slowing or stalled
  • active127
  • slowing2
  • stalled2
  • deprecated2
verify before adopting