---
title: "Create an eval suite (author-only, does not run)"
method: POST
path: "/projects/{projectId}/eval-suites"
tags: ["Eval runs"]
---

# Create an eval suite (author-only, does not run)

`POST /projects/{projectId}/eval-suites`

Creates a runnable eval suite — the suite record plus its test cases — and responds `201` **synchronously**, WITHOUT executing anything. Use this to author a suite, then run it later with `POST /eval-runs` (passing the returned `suiteId`).

This is distinct from `POST /eval-runs`, which creates a run and **detaches execution**, responding `202` with a `runId`. There is no concurrency cap here (no run is started).

The body uses an ergonomic authoring shape: a suite-level default `model` (and optional `provider`) applies to every test unless the test overrides it; `provider` is derived from a `provider/model` id when neither is supplied. Each test's case body is an ordered `steps` array (prompt / toolCall / interact / assert).

Guest callers are denied (suite creation is a write).

## Path parameters

- `projectId` string, required

## Request body

- EvalSuiteCreateRequest — Author-only suite-create body. A suite-level default `model` (and optional `provider`) applies to every test unless the test overrides it.
  - `name` string, required — Suite name.
  - `description` string
  - `serverIds` string[], required — Servers (by canonical project ID) the suite's cases run against.
  - `serverNames` string[] — Optional display names, parallel to `serverIds`.
  - `model` string, required — Suite-level default model id (e.g. `anthropic/claude-haiku-4.5`). Used for any test that omits `model`.
  - `provider` string — Optional suite-level default provider. When omitted, the provider is derived from a `provider/model` id.
  - `passCriteria` object
    - `minimumPassRate` number
  - `tags` string[] — Accepted for forward-compat; not persisted today (no-op).
  - `tests` object[], required — Test cases to create in the suite.
    - `title` string, required
    - `steps` EvalTestStep[], required — Ordered test steps (the unified test model). The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check.
      - `id` string, required
      - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required
      - `prompt` string — User message (`kind: prompt`).
      - `serverName` string — Server that owns the tool (`kind: toolCall`).
      - `toolName` string — Tool name (`kind: toolCall` / `interact`).
      - `arguments` object — Tool-call arguments (`kind: toolCall`).
      - `action` object — Widget action (`kind: interact`).
      - `assertion` object — Predicate or widget assertion (`kind: assert`).
    - `runs` integer — Iterations to execute for this case. Defaults to 1.
    - `model` string — Per-case model override. Defaults to the suite-level `model`.
    - `provider` string — Per-case provider override.
    - `expectedOutput` string
    - `isNegativeTest` boolean — When `true`, the case passes if NO tools are called.
    - `scenario` string
    - `advancedConfig` object — Optional `system`, `temperature`, `toolChoice` overrides.

## Response `201`

The suite was created.

- EvalSuiteCreated
  - `suiteId` string, required
  - `name` string, required
  - `servers` object[] — The servers attached to the suite. `name` is present when supplied.
    - `id` string, required
    - `name` string
  - `caseUpsert` object, required — Per-case create outcomes. Partial failures don't abort the suite; an all-failed new suite is rejected with VALIDATION_ERROR.
    - `committed` object[]
      - `id` string
      - `name` string
    - `failed` object[]
      - `id` string
      - `name` string
      - `error` string

## Other responses

- `400` — Malformed body or parameters.
- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.
- `502` — Could not connect to the target MCP server.

---

[API](https://skmtc.net/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.net/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/versions/75f218486b1d/schema)