latestOpenAPI 3.1.0raw.githubusercontent.com2026-08-16142191531.5 KB

f338eb09898f

Eval runs

Compare a run against a baseline

Compare this run against a baseline run: per-case status (regressed, fixed, new_case, removed_case, changed), per-scorer pass-rate and mean deltas from the evaluation contract, and whether the evaluation config changed. Omit baseRunId to compare against the nearest earlier completed run in the same suite. Returns 404 with details.reason = BASELINE_NOT_FOUND when there is no comparable predecessor — that is an incomplete comparison, not a failing one. A scorer whose definitionChanged is true was graded by a different definition on each side, so its delta is not a regression.

get/projects/{projectId}/eval-runs/{runId}/compare

Path parameters

projectIdstring required

ID of the hosted project that contains the server.

runIdstring required

Eval run ID, as returned by POST /eval-runs.

Query parameters

baseRunIdstring

Run ID to compare against. Omit to use the nearest earlier completed run in the same suite.

Response

The comparison.