---
title: "Compare a run against a baseline"
method: GET
path: "/projects/{projectId}/eval-runs/{runId}/compare"
tags: ["Eval runs"]
---

# Compare a run against a baseline

`GET /projects/{projectId}/eval-runs/{runId}/compare`

Compare this run against a baseline run: per-case status (`regressed`, `fixed`, `new_case`, `removed_case`, `changed`), per-scorer pass-rate and mean deltas from the evaluation contract, and whether the evaluation config changed. Omit `baseRunId` to compare against the nearest earlier **completed** run in the same suite. Returns `404` with `details.reason` = `BASELINE_NOT_FOUND` when there is no comparable predecessor — that is an incomplete comparison, not a failing one. A scorer whose `definitionChanged` is `true` was graded by a different definition on each side, so its delta is not a regression.

## Path parameters

- `projectId` string, required
- `runId` string, required

## Query parameters

- `baseRunId` string

## Response `200`

The comparison.

- EvalRunCompare
  - `suite` object, required
    - `id` string, required
    - `name` string, required
  - `baseline` object, required
    - `policy` 'previous_completed' | 'run', required
    - `baseRunId` string, required
  - `baseRun` EvalRunCompareSide, required
    - `id` string, required
    - `runNumber` integer, required
    - `result` string, required
    - `createdAt` integer, required
    - `completedAt` integer, nullable, required
    - `summary` object, nullable, required
      - `total` integer, required
      - `passed` integer, required
      - `failed` integer, required
      - `passRate` number, required
  - `compareRun` EvalRunCompareSide, required
    - `id` string, required
    - `runNumber` integer, required
    - `result` string, required
    - `createdAt` integer, required
    - `completedAt` integer, nullable, required
    - `summary` object, nullable, required
      - `total` integer, required
      - `passed` integer, required
      - `failed` integer, required
      - `passRate` number, required
  - `passSummary` object, required — Run-summary counters. Named `passSummary`, not `scores`, so it cannot be confused with `scoreContract` — the two answer different questions.
    - `passRatePercent` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `total` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `passed` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `failed` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
  - `metrics` object, required
    - `wallDurationMs` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `totalTokens` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
    - `estimatedCostUsd` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
      - `base` number, nullable, required
      - `compare` number, nullable, required
      - `delta` number, nullable, required
      - `percentDelta` number, nullable, required
  - `scoreContract` object, required
    - `base` ScoreContractSide, required
      - `evaluationConfigHash` string, nullable, required
      - `scoreIntegrity` 'valid' | 'invalid' | 'null', nullable, required — `null` means no verdict was produced. A gate must treat it exactly like `invalid` — absent evidence is not valid evidence.
      - `scoredIterations` integer, required
      - `quarantinedIterations` integer, required — Iterations with at least one row that failed to verify at ingest. Counted here, and excluded from every rate.
    - `compare` ScoreContractSide, required
      - `evaluationConfigHash` string, nullable, required
      - `scoreIntegrity` 'valid' | 'invalid' | 'null', nullable, required — `null` means no verdict was produced. A gate must treat it exactly like `invalid` — absent evidence is not valid evidence.
      - `scoredIterations` integer, required
      - `quarantinedIterations` integer, required — Iterations with at least one row that failed to verify at ingest. Counted here, and excluded from every rate.
    - `evaluationConfigChanged` boolean, required
    - `scorers` ScoreContractScorer[], required
      - `scorerId` string, required
      - `gating` boolean, required
      - `deterministic` boolean, required
      - `definitionChanged` boolean, required — The same scorer id was graded under a different definition hash on each side. Its delta is NOT a regression — the two runs did not measure the same thing.
      - `passRate` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `meanValue` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
      - `errorCount` object, required
        - `base` integer, required
        - `compare` integer, required
  - `cases` EvalRunCompareCase[], required
    - `caseKey` string, required
    - `title` string, required
    - `status` 'unchanged_passed' | 'unchanged_failed' | 'regressed' | 'fixed' | 'new_case' | 'removed_case' | 'changed', required
    - `configChanged` boolean, required — The scenario's own config (prompt, steps, expectations) changed.
    - `evaluationConfigChanged` boolean, required — This case's evaluation config changed.
    - `scoreDeltas` CaseScoreDelta[], required
      - `scorerId` string, required
      - `gating` boolean, required
      - `deterministic` boolean, required
      - `definitionChanged` boolean, required
      - `base` CaseScoreSide, required
        - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required
        - `value` number, nullable, required
        - `passed` boolean, nullable, required
      - `compare` CaseScoreSide, required
        - `status` 'scored' | 'error' | 'skipped' | 'not_applicable', required
        - `value` number, nullable, required
        - `passed` boolean, nullable, required
      - `value` NumericDiff, required — A base/compare pair with its delta. Rate-valued instances are FRACTIONS unless the field name ends in `Percent`.
        - `base` number, nullable, required
        - `compare` number, nullable, required
        - `delta` number, nullable, required
        - `percentDelta` number, nullable, required
    - `base` EvalRunCompareCaseSide, required
      - `outcome` 'passed' | 'failed' | 'absent', required
      - `iterationIds` string[], required
      - `representativeIterationId` string, nullable, required
      - `error` string, nullable, required
    - `compare` EvalRunCompareCaseSide, required
      - `outcome` 'passed' | 'failed' | 'absent', required
      - `iterationIds` string[], required
      - `representativeIterationId` string, nullable, required
      - `error` string, nullable, required

## Other responses

- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.

---

[API](https://skmtc.net/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.net/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/revisions/f338eb09898f/schema)