---
title: "Create an eval run (async)"
method: POST
path: "/projects/{projectId}/eval-runs"
tags: ["Eval runs"]
---

# Create an eval run (async)

`POST /projects/{projectId}/eval-runs`

Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`.

A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`.

For a suite with attached project environments, pass `environmentId` to choose which one the run uses; the `202` echoes the resolved `environment` triple, and `GET /eval-runs/{runId}` reports the same triple for the life of the run.

## Path parameters

- `projectId` string, required

## Request body

- union — Two valid shapes: `suiteId` (rerun an existing suite, optionally upserting inline `tests` into it) or `suiteName` + `tests` + `serverIds` (create a new suite and run it). Inline `tests` alone — without a `suiteId` or a `suiteName` — are rejected with `VALIDATION_ERROR`. `environmentId` requires `suiteId`: an environment is launchable only through a suite that has it attached (`environmentIds`, set via `PATCH /eval-suites/{suiteId}`), so an environment run on a not-yet-created suite could never be satisfied. `environmentId` and `serverIds` are mutually exclusive.
  - object
    - `suiteId` string, required — Existing suite to rerun. A bare `suiteId` with no `tests` reruns the suite exactly as configured.
    - `suiteName` string — Name for a new suite. Required (non-empty) when no `suiteId` is given.
    - `suiteDescription` string
    - `tests` EvalTestCase[] — Inline test cases to upsert into the suite before running.
      - `title` string, required
      - `steps` EvalTestStep[], required — Ordered test steps. The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check.
        - `id` string, required
        - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required
        - `prompt` string — User message (`kind: prompt`).
        - `serverName` string — Server that owns the tool (`kind: toolCall`).
        - `toolName` string — Tool name (`kind: toolCall` / `interact`).
        - `arguments` object — Tool-call arguments (`kind: toolCall`).
        - `action` object — Widget action (`kind: interact`).
        - `assertion` object — Predicate or widget assertion (`kind: assert`).
      - `runs` integer, required — Iterations to execute for this case.
      - `model` string, required — Model ID. Hosted-catalog ids use `provider/name` form (e.g. `anthropic/claude-haiku-4.5`) and run on org credits; provider-native ids require a matching `modelApiKeys` entry (BYOK). Unknown models are rejected with VALIDATION_ERROR at create time.
      - `provider` string, required — Model provider, e.g. `anthropic`, `openai`.
      - `isNegativeTest` boolean — When `true`, the case passes if NO tools are called.
      - `expectedOutput` string
      - `advancedConfig` object — Optional `system`, `temperature`, `toolChoice` overrides.
    - `serverIds` string[] — Servers (by ID) the run connects to. Required when creating a new suite; optional on reruns — when omitted, the run connects the suite's saved server selection (the set its snapshot references). A rerun of a suite with no saved selection is rejected with `VALIDATION_ERROR` (`details.reason: "NO_SAVED_SERVER_SELECTION"`). Rejected outright for a suite with attached environments (`details.reason: "ENVIRONMENT_SERVERS_NOT_OVERRIDABLE"`): the environment supplies a closed set that a server override cannot change, so accepting one would connect a different set than the run is stamped with.
    - `modelApiKeys` object — Optional per-provider model API keys (e.g. `{ "anthropic": "sk-ant-…" }`). Falls back to your organization's configured providers when omitted.
    - `notes` string
    - `passCriteria` object
      - `minimumPassRate` number
    - `iterationOverride` integer — Override the per-case `runs` count for this run only.
    - `environmentId` string — Run against one of the suite's attached project environments. Requires `suiteId`, and must be a member of that suite's `environmentIds` — otherwise `400` with `details.reason: "ENVIRONMENT_NOT_ATTACHED"`, raised before any case is authored or any server connected. Omission is meaningful: a suite with no attached environments runs legacy; a suite with exactly ONE attached environment runs against it automatically (the response's `environment` says which); a suite with several returns `400` with `details.reason: "ENVIRONMENT_REQUIRED"`, naming the candidates. The environment supplies the closed server set (so `serverIds` is not required, and is rejected alongside it), and the run is pinned to the revision resolved at launch — if the environment changes in between, the run is rejected with `409` rather than executing against a different configuration.
  - object
    - `tests` EvalTestCase[], required — Inline test cases to upsert into the suite before running.
      - `title` string, required
      - `steps` EvalTestStep[], required — Ordered test steps. The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check.
        - `id` string, required
        - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required
        - `prompt` string — User message (`kind: prompt`).
        - `serverName` string — Server that owns the tool (`kind: toolCall`).
        - `toolName` string — Tool name (`kind: toolCall` / `interact`).
        - `arguments` object — Tool-call arguments (`kind: toolCall`).
        - `action` object — Widget action (`kind: interact`).
        - `assertion` object — Predicate or widget assertion (`kind: assert`).
      - `runs` integer, required — Iterations to execute for this case.
      - `model` string, required — Model ID. Hosted-catalog ids use `provider/name` form (e.g. `anthropic/claude-haiku-4.5`) and run on org credits; provider-native ids require a matching `modelApiKeys` entry (BYOK). Unknown models are rejected with VALIDATION_ERROR at create time.
      - `provider` string, required — Model provider, e.g. `anthropic`, `openai`.
      - `isNegativeTest` boolean — When `true`, the case passes if NO tools are called.
      - `expectedOutput` string
      - `advancedConfig` object — Optional `system`, `temperature`, `toolChoice` overrides.
    - `suiteId` string — Existing suite to rerun. A bare `suiteId` with no `tests` reruns the suite exactly as configured.
    - `suiteName` string, required — Name for a new suite. Required (non-empty) when no `suiteId` is given.
    - `suiteDescription` string
    - `serverIds` string[], required — Servers (by ID) the run connects to. Required when creating a new suite; optional on reruns — when omitted, the run connects the suite's saved server selection (the set its snapshot references). A rerun of a suite with no saved selection is rejected with `VALIDATION_ERROR` (`details.reason: "NO_SAVED_SERVER_SELECTION"`). Rejected outright for a suite with attached environments (`details.reason: "ENVIRONMENT_SERVERS_NOT_OVERRIDABLE"`): the environment supplies a closed set that a server override cannot change, so accepting one would connect a different set than the run is stamped with.
    - `modelApiKeys` object — Optional per-provider model API keys (e.g. `{ "anthropic": "sk-ant-…" }`). Falls back to your organization's configured providers when omitted.
    - `notes` string
    - `passCriteria` object
      - `minimumPassRate` number
    - `iterationOverride` integer — Override the per-case `runs` count for this run only.
    - `environmentId` string — Run against one of the suite's attached project environments. Requires `suiteId`, and must be a member of that suite's `environmentIds` — otherwise `400` with `details.reason: "ENVIRONMENT_NOT_ATTACHED"`, raised before any case is authored or any server connected. Omission is meaningful: a suite with no attached environments runs legacy; a suite with exactly ONE attached environment runs against it automatically (the response's `environment` says which); a suite with several returns `400` with `details.reason: "ENVIRONMENT_REQUIRED"`, naming the candidates. The environment supplies the closed server set (so `serverIds` is not required, and is rejected alongside it), and the run is pinned to the revision resolved at launch — if the environment changes in between, the run is rejected with `409` rather than executing against a different configuration.

## Response `202`

Run created; execution continues in the background.

- EvalRunCreated
  - `runId` string, required
  - `suiteId` string, required
  - `status` 'running', required
  - `servers` object[] — The servers the run connects to — explicit or derived from the suite's saved selection. `name` is present when known (always, on the derived path).
    - `id` string, required
    - `name` string
  - `caseUpsert` object, required — Per-case upsert outcomes for inline `tests`. Partial failures don't abort the run.
    - `committed` object[]
      - `id` string
      - `name` string
    - `failed` object[]
      - `id` string
      - `name` string
      - `error` string
  - `environment` EvalRunEnvironment, nullable — The project environment a run is pinned to, at the revision resolved when it launched. `null` for a legacy run that used the suite's saved server selection.
    - `id` string, required
    - `name` string, nullable
    - `revision` integer, nullable — The environment revision the run executed against.

## Other responses

- `400` — Malformed body or parameters.
- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.
- `502` — Could not connect to the target MCP server.
- `504` — The target MCP server connected but didn't respond in time.

---

[API](https://skmtc.net/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.net/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/versions/75f218486b1d/schema)