Experiment Management in Kavach Studio

Overview

This tutorial walks through the complete Studio experiment workflow. An experiment groups immutable candidates that vary prompts, model versions, datasets, providers, or runtime parameters while preserving the evidence needed for review and recommendation.

Experiment
  → Candidates
  → Evaluation Runs
  → Comparison
  → Leaderboard
  → Recommendation

Before you start

Start the REST API and Kavach Studio, then open http://localhost:3000/experiments. For a local seeded environment, the API demo data includes experiments, governed registry assets, candidates, completed runs, evaluation metrics, and leaderboard entries.

Candidates reference governed prompt, model, and dataset versions. Those assets must be available in the corresponding registries before a candidate can be added.

Experiment Inventory

The Experiments page is a paginated inventory. It shows the experiment name, status, owner, last modified time, and an action to open the detailed view. Use the status filter to focus on draft, running, completed, failed, or archived experiments.

Create Experiment

Select Create Experiment, provide a name and objective, and submit the form. New experiments begin in DRAFT. The detail page opens after creation so candidates can be registered before execution.

curl -X POST http://localhost:8000/api/v1/experiments \
  -H "content-type: application/json" \
  -d '{
    "name": "Customer Support RAG Evaluation",
    "description": "Compare governed support configurations.",
    "metadata": {"owner": "studio-demo"}
  }'

Overview

The Overview tab summarizes experiment health with candidate count, evaluation run count, success rate, and best leaderboard score. It also shows ownership, created and last-modified timestamps, execution health, metadata, and the latest outcome.

Candidates

Add candidates only from backend-driven registry selections. The form provides governed prompt, model, dataset, and evaluation-provider versions; free-form version strings are not accepted by the Studio workflow.

Each candidate records:

• candidate and experiment identity
• prompt, model, and dataset identity plus version
• evaluation provider
• temperature, top-p, and max-tokens runtime parameters
• metadata and creation timestamp

Candidates are immutable comparison units. To change a governed configuration, register a new candidate instead of modifying the existing one.

Evaluation Runs

Select Start Experiment to evaluate all registered candidates. The Evaluation Runs tab shows each run ID, candidate, status, start time, and duration. Only completed runs provide evaluation evidence for comparison and leaderboard generation; failed runs remain visible for operational review.

Candidates Comparison

The Comparison tab is the detailed candidate review surface. Choose aBaseline Candidate and a different Comparison Candidate. Comparison remains disabled until both selections are different.

The configuration comparison displays these fields side by side:

• prompt identity and version
• model identity and version
• dataset identity and version
• evaluation provider
• temperature, top-p, and max-tokens

Changed configuration rows use neutral theme-aware highlighting. The metric table displays only metrics returned by the backend, with baseline value, comparison value, and delta. Missing metrics are shown as unavailable rather than invented in the browser.

Known directional metrics use semantic colors: green indicates improvement, red indicates regression, and neutral indicates no change or an unknown direction. Higher quality scores are better; lower cost, latency, and hallucination score are better.

Leaderboard

The Leaderboard tab displays the latest ranked snapshot, including rank, candidate, overall score, latency, cost, and ranking reason. A leaderboard is evidence for recommendation; it does not deploy or promote a candidate. Use View in Ontology to inspect the leaderboard and its candidate, run, and evaluation relationships in the graph.

REST & MCP Access

Studio uses the same read contracts that are available to REST and MCP clients:

# Experiment detail data
curl http://localhost:8000/api/v1/experiments/{experiment_id}
curl http://localhost:8000/api/v1/experiments/{experiment_id}/candidates
curl http://localhost:8000/api/v1/experiments/{experiment_id}/runs

# Exact candidate comparison
curl "http://localhost:8000/api/v1/experiments/{experiment_id}/comparison?baseline_candidate_id={baseline_id}&comparison_candidate_id={comparison_id}"

# Latest leaderboard
curl http://localhost:8000/api/v1/experiments/{experiment_id}/leaderboard

from kavach import create_mcp_server

server = create_mcp_server()

server.call_tool("experiment.candidates", {"experiment_id": "experiment-1"})
server.call_tool("experiment.runs", {"experiment_id": "experiment-1"})
server.call_tool(
    "experiment.compare_candidates",
    {
        "experiment_id": "experiment-1",
        "baseline_candidate_id": "candidate-a",
        "comparison_candidate_id": "candidate-b",
    },
)

Operational Notes

• Use Refresh after adding candidates or completing runs to reload the experiment, candidates, runs, leaderboard, and comparison data.
• A comparison requires two different candidates with completed evaluation runs.
• Candidate configuration is sourced from governed registries and should be treated as immutable evidence.
• Leaderboard recommendations are advisory and do not perform deployment or promotion.