KAVACHKAVACH
v2.0.1

Job Execution Control Plane

The Job Execution Control Plane runs synchronous and asynchronous governance workloads with durable lifecycle tracking, idempotency and retry control.

Job Lifecycle

Every job moves through an explicit state machine. Transitions are recorded as audit events, so the full history of a job is always recoverable.

KAVACH / governed event stream
job / execution-2048event order
  1. 01
    accepted

    The idempotency key resolves to one durable job identity.

  2. 02
    claimed

    A worker claims the queued job and starts a renewable lease.

  3. 03
    running

    The handler executes while attempts and lease state remain visible.

  4. 04
    committed

    The result is durably written before the worker acknowledges completion.

governed outcome

SUCCEEDED

Result committed · audit trail retained · retries remain attributable

  • QUEUED — accepted and waiting for a worker.
  • RUNNING — claimed by a worker and executing.
  • SUCCEEDED — completed and results committed.
  • FAILED — terminated after exhausting retries.
  • CANCELLED — explicitly cancelled by an operator or policy.

Idempotent Submission

Submissions carry an idempotency key. Re-submitting the same key returns the existing job rather than creating a duplicate, which makes client retries and at-least-once delivery safe.

http
POST /api/v1/jobs
Idempotency-Key: submit-execution-2048

{
  "type": "evaluation",
  "payload": { "dataset_id": "support-agent", "policy_id": "production-ai-policy" }
}

// 200 OK (returns the same job on repeat)
{ "job_id": "job-5521", "status": "QUEUED" }

Retries

Failed jobs are retried according to a configurable backoff policy. Each attempt is recorded with its error, so operators can distinguish a transient failure from a systematic one. Non-retryable errors move a job directly to FAILED.

Worker Lifecycle

Workers claim queued jobs using a lease. A worker must renew its lease while running; if it stops renewing — because it crashed or was rescheduled — the lease expires and the job becomes eligible for another worker. This prevents jobs from being lost when a worker dies mid-flight.

Interrupted execution

Jobs whose lease expires while RUNNING are flagged as interrupted. They are re-queued only when the job type declares itself safe to retry, protecting against duplicate side effects.

Replay Worker Dispatch

The standard worker runtime consumes REPLAY_EXECUTION, REPLAY_EVALUATION, EVALUATION, and EXPERIMENT jobs. A replay execution commits its produced execution identity before it queues evaluation, so worker redelivery cannot create another Replay or replay execution.

Run the worker independently in local development, or set KAVACH_RUN_REPLAY_WORKER=true to run it beside the API:

bash
uv run python -m kavach.workers.replay_worker_runtime

Cancellation is checked at safe handler boundaries. If a process stops, its lease expires and eligible work can be claimed again without duplicating durable identities.

Persistence

Job records, attempts and state transitions are stored in the relational backend. Because state is durable, the control plane can restart at any time and resume tracking in-flight work without loss.

Operational Controls

  • Cancel a queued or running job.
  • Pause and resume a job queue by type.
  • Inspect attempts, leases and errors for any job.
  • Requeue interrupted jobs after investigation.