Sign in
OpenAPI 3.1.0raw.githubusercontent.com2026-08-19150216600.1 KB

ece7d99ceaf3

Eval runs

Launch one run per target as a group (async)

Launches ONE run per target — attached project environments, or attached named hosts — under a single server-minted runGroupId, then detaches execution and responds 202 with a per-target receipt.

This is the only endpoint with grouped-launch semantics. POST /eval-runs also accepts a runGroupId, but purely as a display label: it groups sibling rows in the UI and gives N separate launches no group treatment. Here the server bounds the fan-out (max 10 targets), validates EVERY target before launching any of them (an unattached target, a heterogeneous target list, or a target whose host selects an unavailable harness returns 400 with zero runs started), and holds exactly ONE organization concurrency slot for the whole group — released only when the last sibling finishes.

Targets fan out along ONE axis: all environmentId or all namedHostId, never a mix. Duplicates are deduplicated by id, preserving order.

A per-target failure does NOT abort its siblings, so read outcome rather than treating the 202 as "everything started": started means every target launched, partial means some did, failed means none did.

Send Idempotency-Key (or idempotencyKey) to make the launch replayable: the group id is DERIVED from the key, and each target carries a run key derived from that, so a retry after a crash mid-launch returns the original run ids instead of double-launching the targets that already started.

post/projects/{projectId}/eval-run-groups

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

Request body

suiteIdstring required

The suite every target runs.

iterationOverrideinteger

Override the per-case runs count for every run in the group.

caseIdsstring[]

Narrow every run in the group to these suite cases.

matchOptionsOverrideobject

Match options for every run in the group. Same public-or-internal vocabulary as POST /eval-runs.

skillsOverride'exclude'

Run every target as the "without skills" arm.

notesstring
idempotencyKeystring

Makes the whole launch replayable. The runGroupId is DERIVED from this key and each target's run key is derived from that, so a retry after a crash mid-launch returns the original run ids rather than double-launching the targets that already started. The Idempotency-Key header carries the same value and wins over this field.

Response

Group accepted. Each target's outcome is in targets; execution continues in the background.

runGroupIdstring required

Server-minted. Every sibling run carries it, and the hosted UI collapses them into one row.

suiteIdstring required
outcome'started' | 'partial' | 'failed' required

started — every target launched; partial — some did and some did not; failed — none did (still a 202: the group itself was valid, and the per-target reasons are in targets).

startedCountinteger required
failedCountinteger required
runIdstring

Mirror of the FIRST started run, so readers written against the single-run receipt keep working. Absent when nothing started. Read targets instead — this describes one run out of several.

statusstring

See runId.

caseUpsertobject

See runId.