v4

latestOpenAPI 3.1.02026-08-01207318738.7 KB

AI Eval

Start an eval run

Create and start a new run against an existing prompt set. The run enqueues run_config.repeat_count agentic jobs per prompt (default 1) and begins executing immediately. Returns the newly created run with its initial per-execution result rows.

post/api/v1/ai/eval/runs

Request body

descriptionstring nullable

Optional human-readable description for the run. Pass null to clear (or omit). Max 1024 characters.

prompt_set_idstring uuid required

The prompt set to execute.

Example request

{
  "description": "Re-running after switching to gpt-4o for query generation",
  "prompt_set_id": "550e8400-e29b-41d4-a716-446655440000",
  "run_config": {
    "branch_id": "440e8400-e29b-41d4-a716-446655440006",
    "repeat_count": 1
  }
}

Response

Run created and jobs enqueued.

job_countinteger required

Number of agentic jobs created for this run (one per prompt execution — prompts × repeat count — that fanned out successfully). Enqueue onto the work queue happens after creation and is best-effort, so this count reflects jobs created, not necessarily those successfully enqueued.

Example response

{
  "job_count": 12,
  "run": {
    "created_at": "2025-01-15T10:00:00.000Z",
    "id": "660e8400-e29b-41d4-a716-446655440001",
    "model_id": "880e8400-e29b-41d4-a716-446655440003",
    "prompt_set_id": "550e8400-e29b-41d4-a716-446655440000",
    "repeat_count": 1,
    "results": [
      {
        "agentic_job": {
          "conversation_id": "770e8400-e29b-41d4-a716-446655440002",
          "id": "990e8400-e29b-41d4-a716-446655440004",
          "state": "COMPLETE"
        },
        "ai_timing_ms": 4121,
        "cost": 0.0021,
        "eval_prompt_id": "bb0e8400-e29b-41d4-a716-446655440007",
        "expectation": "The top product by revenue should be Aniseed Syrup.",
        "id": "aa0e8400-e29b-41d4-a716-446655440005",
        "prompt": "What are the top 5 products by revenue?",
        "query_count": 4,
        "query_timing_ms": 1800,
        "score": 0.9,
        "scoring_cost": 0.0004,
        "timing_ms": 4321,
        "tool_timing_ms": 200
      }
    ],
    "run_number": 3,
    "status": "RUNNING"
  }
}