---
title: "Generate eval cases from the suite's tools"
method: POST
path: "/projects/{projectId}/eval-suites/{suiteId}/cases/generate"
tags: ["Eval runs"]
---

# Generate eval cases from the suite's tools

`POST /projects/{projectId}/eval-suites/{suiteId}/cases/generate`

Discovers the suite's server tools over a live MCP connection, generates cases against them, and persists them — the only edit route that connects to a server, and the only one that SPENDS ORG CREDITS. Synchronous: connect, generate, persist, disconnect, respond.

An environment-based suite generates against that environment's closed server set, so the cases match the tools its runs will actually see.

Pass `x-mcpjam-idempotency-key` to make a retry safe: drafts are recorded backend-side BEFORE any case is persisted, so a replay reuses them instead of spending credits again, and each case is persisted under a derived per-item key so the loop is resumable.

## Path parameters

- `projectId` string, required
- `suiteId` string, required

## Headers

- `x-mcpjam-idempotency-key` string

## Request body

- EvalCaseGenerateRequest — AI-generate cases from the suite's server tools and persist them. SPENDS ORG CREDITS.
  - `mode` 'normal' | 'negative' — Superseded by `caseMix` when that is present.
  - `servers` string[] — Server ids or names to discover tools from. Ignored when the suite is environment-based.
  - `environmentId` string — Discover tools from this attached environment's closed server set, so generated cases are written against the tools the suite's runs will actually see.
  - `caseModels` object[]
    - `model` string, required
    - `provider` string
  - `caseMix` object — Per-bucket case counts. Omitted buckets inherit the default mix; the backend bounds each bucket and the total.
    - `simple` integer
    - `multiTool` integer
    - `multiTurn` integer
    - `complex` integer
    - `negative` integer
  - `varyUserStyles` boolean — Condition generated cases on a range of user styles so the queries read like different users wrote them.

## Response `200`

The generated cases.

- EvalCaseGenerated
  - `generationModel` string, required
  - `created` EvalCase[], required
    - `id` string, required
    - `title` string, required
    - `steps` EvalTestStep[], required — Ordered test steps. A `prompt` step is a model turn; a single model-free `toolCall` step is a render-check; `assert` steps hold the expectations.
      - `id` string, required
      - `kind` 'prompt' | 'toolCall' | 'interact' | 'assert', required
      - `prompt` string — User message (`kind: prompt`).
      - `serverName` string — Server that owns the tool (`kind: toolCall`).
      - `toolName` string — Tool name (`kind: toolCall` / `interact`).
      - `arguments` object — Tool-call arguments (`kind: toolCall`).
      - `action` object — Widget action (`kind: interact`).
      - `assertion` object — Predicate or widget assertion (`kind: assert`).
    - `expectedOutput` string
    - `iterations` integer, required
    - `isNegative` boolean, required — When true, the case passes if NO tools are called.
    - `scenario` string
    - `models` object[], required
      - `model` string, required
      - `provider` string
    - `matchOptions` object, nullable
    - `checks` object, nullable
      - `mode` 'inherit' | 'replace' | 'extend'
      - `list` object[]
    - `createdAt` number, nullable
    - `updatedAt` number, nullable
  - `counts` object, required
    - `normal` integer
    - `negative` integer
  - `skipped` object[] — Drafts that were generated but failed to persist. Surfaced rather than silently dropped.

## Other responses

- `400` — Malformed body or parameters.
- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `409` — The resource is not in a state that accepts this write — a stale `expectedRevision`, a duplicate name, or an environment that cannot currently be launched. The request was well-formed; re-read the resource and retry.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.
- `502` — Could not connect to the target MCP server.
- `504` — The target MCP server connected but didn't respond in time.

---

[API](https://skmtc.net/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.net/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/revisions/b8c36aef7731/schema)