Reproducing a failure in a non-deterministic system

The constraint. You cannot make the model deterministic. Temperature zero still varies with batch composition, hardware and silent provider-side updates. So do not try to replay the model — replay everything around it.

1. 01Record every non-deterministic boundary. Model responses, tool results, timestamps, random values, retrieved document IDs. Replay feeds the recorded values back instead of re-calling. The orchestration becomes fully deterministic while the model remains whatever it was on the day. This is the central trick and it is what durable execution engines already implement.
2. 02Pin model versions explicitly. claude-opus-4-7, never a floating alias. Aliases move, and a six-day-old failure becomes unreproducible for a reason that has nothing to do with your code.
3. 03Record resolved inputs, not templates. The exact bytes sent after templating, retrieval and tool-definition assembly. A trace showing the template and the variables separately is not enough — the bug is often in the join.
4. 04Two distinct replay modes, and people conflate them. Exact replay feeds all recorded values and verifies your orchestration logic. Re-execution feeds recorded inputs to a live model and tells you whether a model or prompt change fixed the behaviour. You need both and they answer different questions.
5. 05Version the retrieval index alongside the code. "Same code, same model, different answer" is most often a silently updated index. Without an index version in the record the investigation dead-ends.
6. 06Sample, because full-fidelity capture is expensive. 100% of errors, 100% of user-flagged runs, 1-5% of successes. The success sample is what gives you a baseline to compare the failures against.

Stack. Temporal for the record-and-replay substrate — deterministic replay is its defining property, not a feature. Langfuse or LangSmith for traces, with the caveat that a trace is an observation and not a replay; they are complementary. Resolved prompts to object storage keyed by trace ID, since tracing backends truncate large payloads exactly when you need them.

Where answers fail. Claiming determinism via temperature zero. It is not true, and the architecture that assumes it has no recovery path when it turns out not to be.