Skip to content
awesome-applied-ai
← Design problems

04Retrieval

Freshness and incremental re-embedding

Our corpus changes 2% per day. Full re-indexing takes 14 hours and costs $8,000. Retrieval quality decays visibly between runs and finance wants the bill halved. Redesign the ingestion pipeline.

The constraint. 2% daily change on a corpus that takes 14 hours to rebuild means you are always serving a stale index, and you are re-embedding 98% of documents that did not change. Both problems have the same root: the pipeline is batch-shaped when the data is stream-shaped.

Architecture.

  1. 01Content-addressed embedding. Hash the exact text that feeds the embedding model. Store the hash alongside the vector. On ingest, recompute the hash; if unchanged, skip. This alone converts a 100% job into a 2% job — the stated $8,000 becomes roughly $160.
  2. 02Separate the change feed from the crawl. Change data capture from the source of record — Postgres logical replication, or the platform's webhook — not a nightly re-crawl. The crawl becomes a weekly reconciliation job that catches what CDC missed, not the primary path.
  3. 03Two-tier index. A large, rarely-rebuilt base index plus a small, continuously-updated delta index. Query both, fuse results. Merge the delta into the base on a schedule. This is how you get minute-level freshness without touching the expensive structure.
  4. 04Deletion is the hard part. Vector indexes tombstone rather than delete; recall degrades and memory does not return until compaction. Track the tombstone ratio and trigger compaction on it, not on a calendar.
  5. 05Version the embedding model explicitly. Model upgrades are the one case where you genuinely must re-embed everything. Store the model identifier per vector, run the new model into a shadow index, and cut over on an eval result rather than on completion.

Stack. Postgres logical replication or Debezium for CDC. LanceDB and Turbopuffer handle incremental writes against object storage without a rebuild, which is the structural fit here. Qdrant supports live upserts with background optimization. Airflow or Temporal for the reconciliation job — Temporal if the pipeline needs to survive partial failure and resume, which at this size it does.

Where answers fail. Optimising the batch job — more workers, bigger machines — rather than eliminating the 98% of it that is redundant.