---
title: "Create an eval run (async)"
method: POST
path: "/projects/{projectId}/eval-runs"
tags: ["Eval runs"]
---

# Create an eval run (async)

`POST /projects/{projectId}/eval-runs`

Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`.

A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`.

For a suite with attached project environments, pass `environmentId` to choose which one the run uses; the `202` echoes the resolved `environment` triple, and `GET /eval-runs/{runId}` reports the same triple for the life of the run.

## Path parameters

- `projectId` string, required

## Request body

- union — Two valid shapes: `suiteId` (rerun an existing suite, optionally upserting inline `tests` into it) or `suiteName` + `tests` + `serverIds` (create a new suite and run it). Inline `tests` alone — without a `suiteId` or a `suiteName` — are rejected with `VALIDATION_ERROR`. `environmentId` requires `suiteId`: an environment is launchable only through a suite that has it attached (`environmentIds`, set via `PATCH /eval-suites/{suiteId}`), so an environment run on a not-yet-created suite could never be satisfied. `environmentId` and `serverIds` are mutually exclusive.
  - object
    - `suiteId` string, required — Existing suite to rerun. A bare `suiteId` with no `tests` reruns the suite exactly as configured.
    - `suiteName` string — Name for a new suite. Required (non-empty) when no `suiteId` is given.
    - `suiteDescription` string
    - `tests` EvalTestCase[] — Inline test cases to upsert into the suite before running.
      - `title` string, required
      - `steps` EvalTestStep[], required — Ordered test steps. The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check.
        - `id` string, required
        - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required
        - `prompt` string — User message (`kind: prompt`).
        - `serverName` string — Server that owns the tool (`kind: toolCall`).
        - `toolName` string — Tool name (`kind: toolCall` / `interact`).
        - `arguments` object — Tool-call arguments (`kind: toolCall`).
        - `action` object — Widget action (`kind: interact`).
        - `assertion` object — Predicate or widget assertion (`kind: assert`).
      - `runs` integer, required — Iterations to execute for this case.
      - `model` string, required — Model ID. Hosted-catalog ids use `provider/name` form (e.g. `anthropic/claude-haiku-4.5`) and run on org credits; provider-native ids require a matching `modelApiKeys` entry (BYOK). Unknown models are rejected with VALIDATION_ERROR at create time.
      - `provider` string, required — Model provider, e.g. `anthropic`, `openai`.
      - `isNegativeTest` boolean — When `true`, the case passes if NO tools are called.
      - `expectedOutput` string
      - `advancedConfig` object — Optional `system`, `temperature`, `toolChoice` overrides.
    - `serverIds` string[] — Servers (by ID) the run connects to. Required when creating a new suite; optional on reruns — when omitted, the run connects the suite's saved server selection (the set its snapshot references). A rerun of a suite with no saved selection is rejected with `VALIDATION_ERROR` (`details.reason: "NO_SAVED_SERVER_SELECTION"`). Rejected outright for a suite with attached environments (`details.reason: "ENVIRONMENT_SERVERS_NOT_OVERRIDABLE"`): the environment supplies a closed set that a server override cannot change, so accepting one would connect a different set than the run is stamped with.
    - `modelApiKeys` object — Optional per-provider model API keys (e.g. `{ "anthropic": "sk-ant-…" }`). Falls back to your organization's configured providers when omitted.
    - `notes` string
    - `passCriteria` object
      - `minimumPassRate` number
    - `iterationOverride` integer — Override the per-case `runs` count for this run only.
    - `environmentId` string — Run against one of the suite's attached project environments. Requires `suiteId`, and must be a member of that suite's `environmentIds` — otherwise `400` with `details.reason: "ENVIRONMENT_NOT_ATTACHED"`, raised before any case is authored or any server connected. Omission is meaningful: a suite with no attached environments runs legacy; a suite with exactly ONE attached environment runs against it automatically (the response's `environment` says which); a suite with several returns `400` with `details.reason: "ENVIRONMENT_REQUIRED"`, naming the candidates. The environment supplies the closed server set (so `serverIds` is not required, and is rejected alongside it), and the run is pinned to the revision resolved at launch — if the environment changes in between, the run is rejected with `409` rather than executing against a different configuration.
    - `namedHostId` string — Run against ONE host attached to the suite. The platform snapshots that host's current config onto the run and derives the run's server set from it, so a host launch needs no `serverIds`. Without this, a suite with host attachments runs under the suite's own default host config — the run executes, but the result is attributed to the wrong host. To run SEVERAL attached hosts, use `POST /eval-run-groups` rather than N calls here: it is the surface that bounds the fan-out and meters it as one launch.
    - `caseIds` string[] — Narrow the run to these suite cases. The persisted suite is untouched — this filters the run's snapshot only. Every id must belong to the suite; none matching returns `404`.
    - `matchOptionsOverride` object — One-off tool-call match options for this run only, layered over suite defaults and per-case overrides. Does NOT mutate the suite or its cases. Accepts EITHER the public vocabulary (`toolCallOrder: any|in-order|exact`, `extraToolCalls`, `arguments`) or the internal one (`toolCallOrder: ignore|superset|strict`, `maxExtraToolCalls`, `argumentMatching`). The two are disjoint, so a body can only be one of them; public bodies are normalized server-side.
      - `toolCallOrder` 'any' | 'in-order' | 'exact' | 'ignore' | 'superset' | 'strict' — Public: `any` (order ignored) | `in-order` (expected calls appear in order, extras allowed) | `exact` (exact sequence). Internal equivalents: `ignore` | `superset` | `strict`.
      - `extraToolCalls` union — Public form: `"unlimited"`, or a maximum count of unexpected extra tool calls.
        - 'unlimited'
        - integer
      - `arguments` 'ignore' | 'partial' | 'exact' — Public form of argument-comparison strictness.
      - `maxExtraToolCalls` integer, nullable — Internal form of `extraToolCalls`: `null` is unlimited.
      - `argumentMatching` 'ignore' | 'partial' | 'exact' — Internal form of `arguments`.
    - `skillsOverride` 'exclude' — The "without skills" arm of an A/B comparison: the run pins NO skills from any channel and is marked `skillsExcluded`, so the arm is labelled rather than merely empty. Scoped to skill DELIVERY — a pinned plugin's MCP servers stay connected, because which servers an arm connects is the one variable a skills A/B has to hold fixed.
    - `refreshSnapshot` boolean — **PERSISTS A SUITE MUTATION.** Re-derives and stores the suite's host-config snapshot from this request's server list, so future runs of the suite use it too. Without it a rerun leaves the snapshot frozen, which is what stops newly connected servers from silently contaminating an existing suite. Single-target launches only — it is not accepted on `POST /eval-run-groups`, where last-writer-wins on a frozen snapshot is never what a fan-out means.
    - `runGroupId` string — A LABEL that groups sibling run rows for display. It has NO quota or launch semantics here: N calls carrying one id are still N independent launches, each metered separately. Grouped-launch behaviour lives on `POST /eval-run-groups`, which mints the id itself. Echoed back on the `202`.
    - `idempotencyKey` string — Write-idempotency key. A repeat call with the same key (same actor and suite) returns the EXISTING run instead of creating and billing a second one. The `Idempotency-Key` header carries the same value and WINS over this field — it is the transport-level channel unattended clients control, whereas a body key could be shaped by model output.
  - object
    - `tests` EvalTestCase[], required — Inline test cases to upsert into the suite before running.
      - `title` string, required
      - `steps` EvalTestStep[], required — Ordered test steps. The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check.
        - `id` string, required
        - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required
        - `prompt` string — User message (`kind: prompt`).
        - `serverName` string — Server that owns the tool (`kind: toolCall`).
        - `toolName` string — Tool name (`kind: toolCall` / `interact`).
        - `arguments` object — Tool-call arguments (`kind: toolCall`).
        - `action` object — Widget action (`kind: interact`).
        - `assertion` object — Predicate or widget assertion (`kind: assert`).
      - `runs` integer, required — Iterations to execute for this case.
      - `model` string, required — Model ID. Hosted-catalog ids use `provider/name` form (e.g. `anthropic/claude-haiku-4.5`) and run on org credits; provider-native ids require a matching `modelApiKeys` entry (BYOK). Unknown models are rejected with VALIDATION_ERROR at create time.
      - `provider` string, required — Model provider, e.g. `anthropic`, `openai`.
      - `isNegativeTest` boolean — When `true`, the case passes if NO tools are called.
      - `expectedOutput` string
      - `advancedConfig` object — Optional `system`, `temperature`, `toolChoice` overrides.
    - `suiteId` string — Existing suite to rerun. A bare `suiteId` with no `tests` reruns the suite exactly as configured.
    - `suiteName` string, required — Name for a new suite. Required (non-empty) when no `suiteId` is given.
    - `suiteDescription` string
    - `serverIds` string[], required — Servers (by ID) the run connects to. Required when creating a new suite; optional on reruns — when omitted, the run connects the suite's saved server selection (the set its snapshot references). A rerun of a suite with no saved selection is rejected with `VALIDATION_ERROR` (`details.reason: "NO_SAVED_SERVER_SELECTION"`). Rejected outright for a suite with attached environments (`details.reason: "ENVIRONMENT_SERVERS_NOT_OVERRIDABLE"`): the environment supplies a closed set that a server override cannot change, so accepting one would connect a different set than the run is stamped with.
    - `modelApiKeys` object — Optional per-provider model API keys (e.g. `{ "anthropic": "sk-ant-…" }`). Falls back to your organization's configured providers when omitted.
    - `notes` string
    - `passCriteria` object
      - `minimumPassRate` number
    - `iterationOverride` integer — Override the per-case `runs` count for this run only.
    - `environmentId` string — Run against one of the suite's attached project environments. Requires `suiteId`, and must be a member of that suite's `environmentIds` — otherwise `400` with `details.reason: "ENVIRONMENT_NOT_ATTACHED"`, raised before any case is authored or any server connected. Omission is meaningful: a suite with no attached environments runs legacy; a suite with exactly ONE attached environment runs against it automatically (the response's `environment` says which); a suite with several returns `400` with `details.reason: "ENVIRONMENT_REQUIRED"`, naming the candidates. The environment supplies the closed server set (so `serverIds` is not required, and is rejected alongside it), and the run is pinned to the revision resolved at launch — if the environment changes in between, the run is rejected with `409` rather than executing against a different configuration.
    - `namedHostId` string — Run against ONE host attached to the suite. The platform snapshots that host's current config onto the run and derives the run's server set from it, so a host launch needs no `serverIds`. Without this, a suite with host attachments runs under the suite's own default host config — the run executes, but the result is attributed to the wrong host. To run SEVERAL attached hosts, use `POST /eval-run-groups` rather than N calls here: it is the surface that bounds the fan-out and meters it as one launch.
    - `caseIds` string[] — Narrow the run to these suite cases. The persisted suite is untouched — this filters the run's snapshot only. Every id must belong to the suite; none matching returns `404`.
    - `matchOptionsOverride` object — One-off tool-call match options for this run only, layered over suite defaults and per-case overrides. Does NOT mutate the suite or its cases. Accepts EITHER the public vocabulary (`toolCallOrder: any|in-order|exact`, `extraToolCalls`, `arguments`) or the internal one (`toolCallOrder: ignore|superset|strict`, `maxExtraToolCalls`, `argumentMatching`). The two are disjoint, so a body can only be one of them; public bodies are normalized server-side.
      - `toolCallOrder` 'any' | 'in-order' | 'exact' | 'ignore' | 'superset' | 'strict' — Public: `any` (order ignored) | `in-order` (expected calls appear in order, extras allowed) | `exact` (exact sequence). Internal equivalents: `ignore` | `superset` | `strict`.
      - `extraToolCalls` union — Public form: `"unlimited"`, or a maximum count of unexpected extra tool calls.
        - 'unlimited'
        - integer
      - `arguments` 'ignore' | 'partial' | 'exact' — Public form of argument-comparison strictness.
      - `maxExtraToolCalls` integer, nullable — Internal form of `extraToolCalls`: `null` is unlimited.
      - `argumentMatching` 'ignore' | 'partial' | 'exact' — Internal form of `arguments`.
    - `skillsOverride` 'exclude' — The "without skills" arm of an A/B comparison: the run pins NO skills from any channel and is marked `skillsExcluded`, so the arm is labelled rather than merely empty. Scoped to skill DELIVERY — a pinned plugin's MCP servers stay connected, because which servers an arm connects is the one variable a skills A/B has to hold fixed.
    - `refreshSnapshot` boolean — **PERSISTS A SUITE MUTATION.** Re-derives and stores the suite's host-config snapshot from this request's server list, so future runs of the suite use it too. Without it a rerun leaves the snapshot frozen, which is what stops newly connected servers from silently contaminating an existing suite. Single-target launches only — it is not accepted on `POST /eval-run-groups`, where last-writer-wins on a frozen snapshot is never what a fan-out means.
    - `runGroupId` string — A LABEL that groups sibling run rows for display. It has NO quota or launch semantics here: N calls carrying one id are still N independent launches, each metered separately. Grouped-launch behaviour lives on `POST /eval-run-groups`, which mints the id itself. Echoed back on the `202`.
    - `idempotencyKey` string — Write-idempotency key. A repeat call with the same key (same actor and suite) returns the EXISTING run instead of creating and billing a second one. The `Idempotency-Key` header carries the same value and WINS over this field — it is the transport-level channel unattended clients control, whereas a body key could be shaped by model output.

## Response `202`

Run created; execution continues in the background.

- EvalRunCreated
  - `runId` string, required
  - `suiteId` string, required
  - `status` 'running', required
  - `runGroupId` string — Echo of the request's `runGroupId`, when one was sent. A LABEL only — it groups sibling rows for display and carries no quota or launch semantics.
  - `servers` object[] — The servers the run connects to — explicit or derived from the suite's saved selection. `name` is present when known (always, on the derived path).
    - `id` string, required
    - `name` string
  - `caseUpsert` object, required — Per-case upsert outcomes for inline `tests`. Partial failures don't abort the run.
    - `committed` object[]
      - `id` string
      - `name` string
    - `failed` object[]
      - `id` string
      - `name` string
      - `error` string
  - `environment` EvalRunEnvironment, nullable — The project environment a run is pinned to, at the revision resolved when it launched. `null` for a legacy run that used the suite's saved server selection.
    - `id` string, required
    - `name` string, nullable
    - `revision` integer, nullable — The environment revision the run executed against.

## Other responses

- `400` — Malformed body or parameters.
- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.
- `502` — Could not connect to the target MCP server.
- `504` — The target MCP server connected but didn't respond in time.

---

[API](https://skmtc.net/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.net/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/revisions/ece7d99ceaf3/schema)