latestOpenAPI 3.1.0raw.githubusercontent.com2026-08-21161224637.5 KB

8978a4e735fa

Eval runs

Create an eval run (async)

Creates a suite run from an existing suiteId (rerun) and/or inline tests, then detaches execution and responds 202 immediately with the runId. Validation and quota errors surface on this request; poll GET /eval-runs/{runId} for progress. The run appears live in the hosted UI Runs tab, tagged source: "api".

A bare suiteId with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns 429 with details.reason: "CONCURRENT_RUN_LIMIT".

For a suite with attached project environments, pass environmentId to choose which one the run uses; the 202 echoes the resolved environment triple, and GET /eval-runs/{runId} reports the same triple for the life of the run.

post/projects/{projectId}/eval-runs

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

Request body

Response

Run created; execution continues in the background.

runIdstring required

suiteIdstring required

statusstring required

The run's status. running on a fresh launch; on a replay (see deduped), the existing run's own status, which may already be terminal.

dedupedboolean

Present and true when this request REPLAYED an existing run instead of starting one (an idempotency-key hit, or the short keyless dedupe window). A replayed run is not executed again, so no further credits are spent; read status for what that run actually is. Absent on a fresh launch.

runGroupIdstring

Echo of the request's runGroupId, when one was sent. A LABEL only — it groups sibling rows for display and carries no quota or launch semantics.