v20

OpenAPI 3.1.0raw.githubusercontent.com2026-08-035471202.8 KB
Eval runs

Create an eval suite (author-only, does not run)

Creates a runnable eval suite — the suite record plus its test cases — and responds 201 synchronously, WITHOUT executing anything. Use this to author a suite, then run it later with POST /eval-runs (passing the returned suiteId).

This is distinct from POST /eval-runs, which creates a run and detaches execution, responding 202 with a runId. There is no concurrency cap here (no run is started).

The body uses an ergonomic authoring shape: a suite-level default model (and optional provider) applies to every test unless the test overrides it; provider is derived from a provider/model id when neither is supplied. Each test's case body is an ordered steps array (prompt / toolCall / interact / assert).

Guest callers are denied (suite creation is a write).

post/projects/{projectId}/eval-suites

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

Request body

namestring required

Suite name.

descriptionstring
serverIdsstring[] required

Servers (by canonical project ID) the suite's cases run against.

serverNamesstring[]

Optional display names, parallel to serverIds.

modelstring required

Suite-level default model id (e.g. anthropic/claude-haiku-4.5). Used for any test that omits model.

providerstring

Optional suite-level default provider. When omitted, the provider is derived from a provider/model id.

tagsstring[]

Accepted for forward-compat; not persisted today (no-op).

Response

The suite was created.

suiteIdstring required
namestring required