Replay is not retry: one word that changes the integrity of the evidence

“Just run it again” is one of the most dangerous sentences in an AI system.

It sounds sensible because it borrows intuition from ordinary software operations. A request timed out. A worker died. A network call failed. Retry the same unit of work, make the operation idempotent, and move on. That model is correct for a large class of failures.

But it is incomplete for a system whose output can influence a customer, a claim, a hiring process, a transaction, or a release decision. In those systems, the question is rarely only did it succeed? The question becomes: what exactly was decided, from which inputs, under which configuration, and would the same governed process reach a materially different result today?

Two operations, two questions

A retry answers an operational question: can the original unit of work reach its intended terminal state after a transient interruption? Its ideal outcome is that no one can tell it happened. The same request keeps the same identity and should not produce a second external side effect.

A replay answers an evidentiary question: what happens when we intentionally reproduce a historical execution from the evidence that governed it? Its ideal outcome is not invisibility. A replay should create a distinct execution, preserve its relationship to the source, and expose any meaningful difference.

Why identity matters

If a replay reuses the source execution identity, the evidence becomes ambiguous. Was the result produced before or after a prompt change? Did it use the model version that was originally approved, or the one currently configured? Was an observed regression caused by the system or by a silent change in the test itself?

Kavach treats a Replay as its own governed aggregate. It has its own ID, request reason, tenant scope, frozen configuration hash, execution job, produced execution ID, evaluation job, comparison reference, drift reference, and final result. The source remains immutable. The new execution records lineage back to it.

That design is deliberately less convenient than a “rerun” button. It is also much harder to accidentally lie with.

Frozen does not mean simplistic

“Frozen configuration” is often reduced to a model name and a temperature. In practice, a governed execution may depend on input and state snapshots, artifacts, prompts, models, datasets, policies, an execution adapter, and runtime parameters. If any of those are fetched from their latest version during a replay, the replay has stopped being evidence about the past.

Freezing these references is not an attempt to make systems static. It is an attempt to make change legible. You can still run an experiment against a new prompt or model. You can still evaluate a new policy. But those are different operations, with different hypotheses, than asking whether a historical execution can be reproduced under its recorded conditions.

The baseline is a contract

A replay evaluation needs more than a produced output. It needs a source evaluation that can be compared honestly. A source baseline must belong to the same tenant and source execution, use the same evaluator type and version, and share at least one metric with the replay evaluation.

That sounds strict because it is. Comparing a relevance score from one evaluator to a score from another and calling the difference “drift” is theater. The metric labels may look the same while the scoring rubric, model, or version has changed underneath them.

The practical consequence is useful: when no compatible source evaluation exists, the system should say so. “No compatible baseline” is not a failed comparison. It is a statement that the organization did not retain enough historical evidence to make the comparison defensible.

A better default

Teams should make the distinction explicit in their operating model:

• Use retry for interrupted work where the intended result is unchanged.
• Use replay to reproduce an execution from frozen historical evidence.
• Use an experiment to deliberately vary a governed configuration.
• Use comparison and drift analysis to turn a difference into a reviewable claim.

These are not product labels. They are different epistemic promises. Once a system makes consequential decisions, preserving that distinction is one of the simplest ways to avoid manufacturing confidence after the fact.