v2.0.1

Replay Management

Use Kavach Studio to create a governed reproduction of a historical workflow execution, then follow its durable execution and evaluation evidence through comparison and drift analysis.

What you will do

In this tutorial, you will select one completed historical execution, freeze the configuration that made it replayable, submit a new Replay, and review the produced execution, compatible evaluation baseline, comparison, and drift result.

Historical source
  → frozen Replay configuration
  → replay execution job
  → replay evaluation job
  → baseline comparison + drift
  → immutable Replay result

Replay is not retry

A Replay never reruns or changes the historical source. It creates a new execution from persisted source evidence and records explicit lineage between the two executions.

Before you start

Start the API and Studio, then run the durable worker in a second terminal. In local or development mode, API startup seeds replayable source executions and compatible primary mock@1.0.0 evaluations for this walkthrough.

bash

# Terminal 1
uv run uvicorn kavach.api.app:app --reload

# Terminal 2
uv run python -m kavach.workers.replay_worker_runtime

# Terminal 3
cd console && pnpm dev

Open http://localhost:3000/replays. You need permission to read, create, execute, and evaluate replays in the selected tenant project. The Jobs page should show worker activity as the workflow progresses.

1. Select a source execution

Select Create Replay. Search by execution ID, workflow, or service and keep Replayable only enabled when you want to exclude incomplete historical evidence. The search is tenant-scoped and cursor-paginated; it does not load every execution ID into the browser.

Enter demo-source-execution in the search box.
Select a completed result, such as demo-source-execution-06.
Select Use source.
Enter a clear reason, such as Verify a support workflow after a platform upgrade.

In production, use a source execution that has retained input and state snapshots plus the artifact, prompt, model, dataset, and policy references required by its historical adapter.

2. Validate frozen evidence

Select Validate replayability. Kavach resolves the source again on the server, checks authorization and source evidence, then presents the frozen configuration. Review the workflow version, execution adapter, input and state snapshot references, configuration hash, and expanded runtime references before continuing.

What the configuration hash means

The hash identifies the immutable replay configuration. Worker retries retain this exact configuration; they do not look up the current workflow definition or silently adopt a newer prompt, model, or policy.

3. Create the replay

Select Create replay. The new Replay starts in READY; creation only records the governed request and frozen configuration. It does not run the workflow yet.

The detail page exposes a copyable Replay ID, source execution ID, configuration, and audit context. Save the Replay ID if you will continue through REST or MCP instead of Studio.

4. Submit and monitor execution

On the Replay detail page, provide an action reason and select Submit Replay. The API creates oneREPLAY_EXECUTION job and moves the Replay through QUEUED and RUNNING. The worker claims the job with a lease, reconstructs a new execution from the frozen configuration, and stores the source-to-produced-execution lineage.

With the supplied worker runtime, successful execution automatically submits the separate REPLAY_EVALUATION job. Open /jobs to inspect both jobs. Its Runs view refreshes every five seconds by default; you can temporarily choose Off, 10s, or 30s.

Watch the Replay status progress through QUEUED → RUNNING → EXECUTION_COMPLETED → EVALUATING → COMPARING → COMPLETED. The exact intermediate transitions can be brief with the local mock evaluator.

5. Evaluate and compare

Think of this step as a before-and-after review. The worker evaluates the new replay execution to produce the after scorecard. It then loads a saved evaluation of the original source execution as thebefore scorecard, called the baseline. Kavach compares those two scorecards; it does not evaluate the source again during this step.

text

Source execution evaluation  = baseline = before
Replay execution evaluation  = candidate = after
Before + after               = comparison and drift

By default, Kavach uses the latest compatible source evaluation. You can instead choose the source primary baseline, or provide an explicit source evaluation ID when a review must use one particular historical scorecard.

A source baseline is compatible only when it has:

the same tenant and source execution;
the same evaluator type and version; and
at least one metric shared with the replay evaluation.

If automatic evaluation is not enabled for your worker deployment, wait for EXECUTION_COMPLETED, open the Evaluation section of the Replay detail page, choose a baseline strategy, and select Evaluate Replay.

If there is no baseline

This means the source may be replayable but it was never evaluated with a comparable evaluator and metric schema. Evaluate the original source first, then create a new Replay or select that saved evaluation explicitly. A retry cannot recreate historical evidence that was never stored.

6. Inspect result and lineage

When the Replay reaches COMPLETED, open the Comparison tab. It shows the compact evidence summary: metric count, improved, regressed, unchanged, new, and removed metrics; overall score delta; drift severity; and the effective drift threshold policy.

Open the Lineage tab to follow persisted references from the source execution to the Replay, replay execution, execution and evaluation jobs, evaluations, comparison, drift, and result. Drag nodes to inspect a dense graph, select a node for its metadata, and use Reset to restore the generated layout. These layout changes are local to the browser.

Troubleshooting and cancellation

No compatible source evaluation is available. Evaluate the source first with the intended provider and metric schema, then create a new Replay. This is historical evidence missing from the source, not a worker retry condition.
Replay remains queued. Confirm the worker process is running and that API and worker share the same Job, Replay, evaluation, and execution-store configuration.
Worker interruption. Let the job lease expire or restart a worker. Durable replay execution identity and evaluation-job idempotency prevent a duplicate produced execution or Replay.
Cancel active work. Select Cancel on the Replay detail page. Queued work is cancelled immediately; running handlers observe the request at safe boundaries. Evidence already committed remains available for review.

REST walkthrough

Studio uses these same APIs. The following commands illustrate a dry-run, creation, submission, and result lookup. Supply your normal tenant and authentication headers in non-development environments.

bash

# Find a bounded page of eligible sources
curl "http://localhost:8000/api/v1/replay-executions/search?query=demo-source-execution&replayable_only=true&limit=20"

# Validate without reserving an idempotency key or creating a Replay
curl -X POST http://localhost:8000/api/v1/replays \
  -H 'content-type: application/json' \
  -d '{
    "source_execution_id": "demo-source-execution-06",
    "idempotency_key": "support-upgrade-replay-001",
    "reason": "Verify support workflow after platform upgrade.",
    "dry_run": true
  }'

# Create the durable Replay request
curl -X POST http://localhost:8000/api/v1/replays \
  -H 'content-type: application/json' \
  -d '{
    "source_execution_id": "demo-source-execution-06",
    "idempotency_key": "support-upgrade-replay-001",
    "reason": "Verify support workflow after platform upgrade.",
    "metadata": {"change_ticket": "CHG-1042"}
  }'

# Submit the execution job, then inspect the completed immutable result
curl -X POST http://localhost:8000/api/v1/replays/{replay_id}/submit \
  -H 'content-type: application/json' \
  -d '{"reason":"Approved replay execution"}'
curl http://localhost:8000/api/v1/replays/{replay_id}/result