KAVACHKAVACH
v2.0.1

Experiment Management

Experiments create controlled comparison spaces for multiple candidates that share a problem definition while varying prompts, model versions, datasets, or runtime parameters.

Why Experiments Exist

Evaluation results on their own answer whether one execution scored well. Experiment management answers the harder governance question: which governed AI configuration should be recommended for a business use case.

An experiment creates a controlled comparison space for multiple candidates that share a problem definition while varying prompts, model versions, datasets, or runtime parameters.

KAVACH / governed event stream
experiment / candidate reviewevent order
  1. 01
    bind immutable candidates

    Each candidate fixes the prompt, model, dataset, provider and runtime configuration.

  2. 02
    run governed evaluations

    Evaluation runs produce comparable evidence for every candidate.

  3. 03
    compare the evidence

    Configuration differences and quality changes remain visible together.

  4. 04
    publish leaderboard

    The selected ranking strategy creates a durable recommendation artifact.

governed outcome

RECOMMENDED

Evidence compared · ranking recorded · deployment remains a separate decision

Core Objects

  • WorkflowExecution — runtime evidence capturing what happened during a workflow execution.
  • EvaluationResult — metric evidence for a single execution. Records scores, metadata, and evaluator identity.
  • EvaluationRun — experiment evidence linking an experiment candidate to a governed evaluation attempt, tracking status and timestamps.
  • Experiment — the orchestration aggregate defining the comparison space and its lifecycle.
  • ExperimentCandidate — an immutable AI configuration binding prompt, model, dataset, evaluation provider, and runtime parameters into one governed unit.
  • Leaderboard — the final experiment-facing governance artifact capturing a snapshot of ranked candidates under a defined ranking strategy.

Experiment Lifecycle

DRAFT
  → RUNNING
       ├──→ FAILED (terminal)
       └──→ COMPLETED
              → ARCHIVED (terminal governance state)

Candidate as Immutable Configuration

A candidate exists so that a ranking decision can be tied to an exact configuration. The candidate should be read as a versioned contract:

  • Prompt identity and version
  • Model identity and version
  • Dataset identity and version
  • Evaluation provider
  • Runtime settings
  • Metadata

Changing any of those values should produce a new candidate record rather than mutating the old one.

Candidate Comparison

Candidate comparison explains what changed between candidates before ranking answers which candidate is preferred. It is useful for reviews because it keeps configuration change and quality change visible together.

Studio Workflow

Kavach Studio provides a complete experiment workflow covering the experiment inventory, governed candidate registration, evaluation runs, field-by-field comparison, and leaderboard review.
The detailed walkthrough is available in the Experiment Management in Kavach Studio tutorial.

Ranking Strategy

Ranking strategies allow the platform to optimize for different objectives without rewriting experiment models. Current strategies include overall score, groundedness, answer relevance, latency, cost, and hallucination-aware ranking.

Leaderboard

The leaderboard is the artifact that downstream systems should consume when they need an experiment recommendation. It is a snapshot, not a live computation.

Experiment
  → Candidates
  → Evaluation Runs
  → Comparison
  → Ranking Strategy
  → Leaderboard
  → Recommendation

Recommendation Is Not Deployment

A recommended candidate is not a deployed candidate. Kavach identifies the highest-ranked candidate and records the reason. Another system may choose to consume that recommendation as one input into promotion or release logic.