Skip to content
awesome-applied-ai

The cycle

Eight stages, in order

What an enterprise system passes through between someone asking for it and someone signing off on it. It is a cycle rather than a pipeline because stage 08 sends you back to stage 01 with better information than you started with. Most systems that fail in production skipped a stage rather than executing one badly.

8 stages · 8 projects to build them yourself

  1. 01

    Frame

    What decision changes when this ships?

    What you compute

    The baseline you have to beat and the unit you will be judged in. Cost per resolved case, minutes saved per reviewer, percent of queue handled without a human.

    Where it goes wrong

    Starting from the model instead of the decision. A system with no named baseline cannot be shown to work, so it gets judged on demos.

  2. 02

    Source

    01 Retrieval

    Where does the data actually live, and what may you do with it?

    What you compute

    Freshness budget, update volume per day, tenancy boundaries, and which fields you are not allowed to send anywhere.

    Where it goes wrong

    Discovering after the pipeline is built that the corpus changes 2% per day, which turns a one-off embedding job into a standing bill.

  3. 03

    Ground

    01 Retrieval

    How does a query become evidence the model can use?

    What you compute

    Index memory budget, chunk boundaries, filter selectivity, and whether you need a graph at all. Almost always the answer is no.

    Where it goes wrong

    Naming a vector database before computing rows x dims x bytes. Quantization moves that number by 30x; the engine moves it by a little.

  4. 04

    Assemble

    03 Caching & Context Optimization

    What goes in the window, in what order, at what price?

    What you compute

    The stable prefix, the cache hit rate you can realistically hold, and the point where stuffing the window beats retrieving into it.

    Where it goes wrong

    Rebuilding the prompt prefix on every call. One reordered block turns a cache hit into a cache write, and the write costs more than the read it replaced.

  5. 05

    Orchestrate

    04 Orchestration & Protocols

    One call, a chain, or several agents?

    What you compute

    Topology, tool surface, budget gates, and what happens when step four fails after step three already charged a customer.

    Where it goes wrong

    Reaching for multiple agents on a task with a single dependency chain. Every handoff is a lossy serialization of context you already had.

  6. 06

    Prove

    05 Evaluation & Observability

    Is the new thing better than the old thing on your data?

    What you compute

    A labelled set drawn from real traffic, the metric that gates release, and the sample size that makes a 2 point difference mean something.

    Where it goes wrong

    Reporting generation quality as evidence retrieval improved. recall@k is the ceiling on everything downstream, and it is measured separately.

  7. 07

    Serve

    03 Caching & Context Optimization

    What does it cost at real concurrency, and how fast is P95?

    What you compute

    Where the cheap path ends and the expensive one starts. Most production systems are a cascade, and the cascade is designed here or discovered in the incident.

    Where it goes wrong

    Sizing on P50. The tail is where the timeouts, the retries and the duplicated spend live.

  8. 08

    Govern

    06 Guardrails & Governance

    Who signs off, what is logged, and how do you replay a bad answer?

    What you compute

    Review queue design, audit trail, retention, and the approval that has to survive a three day pause without losing state.

    Where it goes wrong

    Bolting on a human review step at the end. If a reviewer cannot see why the system answered that way, the queue becomes a rubber stamp.

Build it yourself

8 projects, one stage at a time

None of these need a company, a budget or a cluster. Each one exists to produce a single number you measured on your own data, because that is the difference between having read about a stage and having done it.

01Frame

  • Count your own repository

    An evening

    A script that tokenizes every file in a repo you know well and reports the distribution, the ten biggest files, and what a full read would cost at current prices.

    TeachesToken accounting as a habit. After this you stop guessing whether something fits and start knowing.

    MeasureTotal tokens, and the dollar cost of one full pass.

03Ground

  • Search your own notes two ways

    A weekend

    BM25 over your markdown notes, then embeddings over the same corpus, then both fused. Ten queries you already know the right answer to.

    TeachesWhy lexical search keeps winning on rare terms, and why fusion raises recall without touching precision.

    Measurerecall@10 for each of the three, on your own ten queries.

  • Chunk one genuinely messy PDF

    A weekend

    A parser for a document with tables, footnotes and multi column layout. Compare fixed size splits against structure aware splits.

    TeachesThat chunking is where most retrieval quality is won or lost, and that it is unglamorous parsing work.

    MeasureHow many of the tables survive intact, counted by hand. This number is usually worse than you expect.

04Assemble

  • Watch a prompt cache pay or fail

    An evening

    The same task run twice, once with a stable prefix and once with the blocks shuffled. Log the cached and uncached token counts from the API response.

    TeachesThat caching is a property of prompt construction, not a setting you enable.

    MeasureHit rate and cost delta across fifty calls.

05Orchestrate

  • One agent, three tools

    A weekend

    An MCP server exposing three tools over data you own, then a single agent that uses them. Resist adding a fourth.

    TeachesHow much a tool description costs in the prefix, and how quickly a wide tool surface degrades selection.

    MeasureTokens spent on tool definitions per turn, and selection accuracy over twenty tasks.

06Prove

  • Build a golden set from your own traffic

    Two weekends, and it never really finishes

    Fifty real queries, labelled by you, split into an easy half and a hard half. Then a script that runs any change against them.

    TeachesThat the labelling is the work, and that fifty honest examples beat a thousand synthetic ones.

    MeasurePass rate on the hard half. Track it across every change you make.

07Serve

  • Run a local model and measure it properly

    An evening

    A small model on your own machine, benchmarked at batch size 1 and at concurrency 8, with memory and tokens per second recorded.

    TeachesWhere the latency actually goes, and how far a 7B model gets you on a task you care about.

    MeasureTokens per second at both concurrencies, and peak resident memory.

08Govern

  • Redact before you send

    A weekend

    A pre flight pass that strips identifiers from a request, plus a log that records what was stripped without recording the values.

    TeachesThe shape of every data protection conversation you will have in an enterprise.

    MeasureFalse negatives on a hundred hand checked samples. One miss is the whole story.