---
title: "Start an eval run"
method: POST
path: "/api/v1/ai/eval/runs"
tags: ["AI Eval"]
---

# Start an eval run

`POST /api/v1/ai/eval/runs`

Create and start a new run against an existing prompt set. The run enqueues `run_config.repeat_count` agentic jobs per prompt (default 1) and begins executing immediately. Returns the newly created run with its initial per-execution result rows.

## Request body

- EvalRunsCreateBody
  - `description` string, nullable — Optional human-readable description for the run. Pass `null` to clear (or omit). Max 1024 characters.
  - `prompt_set_id` string, uuid, required — The prompt set to execute.
  - `run_config` object — Per-run configuration. Optional — omit if no overrides.
    - `branch_id` string, uuid — Optional branch ID to run against. Must be a branch of the prompt set's model.
    - `repeat_count` integer — How many times to execute each prompt in the set (defaults to 1). Between 1 and 10; prompts × repeats may not exceed the per-run job limit.

## Response `201`

Run created and jobs enqueued.

- EvalRunsCreateResponse
  - `job_count` integer, required — Number of agentic jobs created for this run (one per prompt execution — prompts × repeat count — that fanned out successfully). Enqueue onto the work queue happens after creation and is best-effort, so this count reflects jobs created, not necessarily those successfully enqueued.
  - `run` EvalRunDetail, required — The newly created run with its initial results.
    - `branch_id` string, uuid, nullable, required — Optional branch ID the run was executed against. Null when run against the main shared model.
    - `branch_name` string, nullable, required — Display name for the branch, if `branch_id` is set.
    - `completed_at` string, nullable, required — ISO 8601 timestamp when the run reached a terminal state.
    - `created_at` string, nullable, required — ISO 8601 timestamp when the run was created.
    - `description` string, nullable, required — Optional human-readable description for the run.
    - `id` string, uuid, required — Unique identifier for the run.
    - `is_archived` boolean, required — Whether the run has been archived.
    - `model_id` string, uuid, required — The shared model this run was executed against.
    - `prompt_set_id` string, uuid, required — The prompt set this run was created from.
    - `repeat_count` integer, required — How many times each prompt in the set was executed. Results carry a `repeat_index` when this is greater than 1.
    - `results` EvalRunResult[], required — Per-execution results for this run (one per prompt, times the repeat count). No guaranteed order — group executions by `eval_prompt_id` and order by `repeat_index`.
      - `agentic_job` EvalRunResultAgenticJob, required
        - `conversation_id` string, uuid, nullable, required — Conversation the agentic job belongs to.
        - `id` string, uuid, required — Agentic job identifier.
        - `state` 'CANCELLED' | 'COMPLETE' | 'DELIVERING' | 'EXECUTING' | 'FAILED' | 'QUEUED', required — Current state of the agentic job that ran this prompt.
      - `ai_timing_ms` integer, nullable, required — Strict main-agent LLM processing time in milliseconds — the measured model-call duration, excluding tool execution and subagent model calls (those count toward `tool_timing_ms`). Shown as "AI time" in the UI. Runs recorded before this was measured fall back to an approximation (`timing_ms` minus tool latency).
      - `cost` number, nullable, required — Total LLM cost (USD) for this prompt, if available.
      - `error_reason` string, nullable, required — Failure reason string for prompts whose underlying job failed.
      - `eval_prompt_id` string, uuid, nullable, required — Snapshot of the prompt id this result was executed for — repeated executions of the same prompt share it, and it survives later prompt deletion. Null on runs created before repeats existed.
      - `expectation` string, nullable, required — The prompt's expectation as of run creation (snapshotted, so later prompt edits don't change past runs), or null when none was set. The analysis judge scores the analysis against it.
      - `id` string, uuid, required — Unique identifier for the run result row.
      - `prompt` string, required — The prompt text that was evaluated.
      - `query_count` integer, nullable, required — Number of warehouse queries the underlying job ran. Null for runs executed before this metric was recorded.
      - `query_timing_ms` integer, nullable, required — Total wall-clock time (milliseconds) the underlying job spent running warehouse queries — a proxy for query execution time. Null for runs executed before this metric was recorded.
      - `repeat_index` integer, nullable, required — 0-based repeat number of this execution within the run (see the run's `repeat_count`). Null on runs created before repeats existed.
      - `score` number, nullable, required — Numeric judge score for this prompt result, if scoring ran.
      - `scoring_cost` number, nullable, required — Total LLM cost (USD) for scoring this prompt result.
      - `timing_ms` integer, nullable, required — Total `/generate` wall-time in milliseconds — LLM processing plus inner-loop tool execution. `ai_timing_ms` and `tool_timing_ms` split this; warehouse query time is separate (`query_timing_ms`).
      - `tool_timing_ms` integer, nullable, required — Inner-loop tool latency in milliseconds — time spent running tools the model invoked (model and field-value lookups, query planning), excluding the warehouse query itself (`query_timing_ms`). Null for runs recorded before per-tool latency was tracked.
    - `run_number` integer, required — Sequential, per-prompt-set run number.
    - `status` 'RUNNING' | 'COMPLETE' | 'CANCELLED', required — Run-level lifecycle. Flips to a terminal state (COMPLETE or CANCELLED) exactly once.

## Other responses

- `400` — Invalid request body.
- `401` — Missing or invalid API key.
- `403` — Insufficient permissions. The caller must have at least the Querier role on the prompt set's model.
- `404` — The prompt set was not found, or `run_config.branch_id` does not match an existing branch in the organization.
- `422` — `run_config.branch_id` does not belong to the prompt set's model, or prompts × `run_config.repeat_count` exceeds the per-run job limit.
- `429` — Per-user active-run cap reached.
- `500` — Run created and jobs enqueued, but it could not be re-read for the response. The run exists — list runs for the prompt set to find it rather than retrying, since a retry starts a duplicate run.
- `503` — AI eval is paused for this organization.

---

[API](https://skmtc.net/omniapp/apis/omni-api.md) · [All operations](https://skmtc.net/omniapp/apis/omni-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/omniapp/omni-api/versions/de7cac8b5983/schema)