---
title: "Run an eval suite on a repository's pull requests"
method: POST
path: "/organizations/{organizationId}/eval-check-repos"
tags: ["Evals"]
---

# Run an eval suite on a repository's pull requests

`POST /organizations/{organizationId}/eval-check-repos`

Connect a repository so every pull request to it runs one eval suite and reports a GitHub check. **Affects everyone who opens a pull request on that repository**, and with `outagePolicy: fail_closed` it can block their merges.

The repository is verified against the organization's GitHub App installation before it is stored. A repository that does not exist and one the App cannot see answer identically, on purpose: answering differently would make this an oracle for private repository names.

Deliberately narrow: retargeting a repository at a different suite, pausing it, and disconnecting are repo-level decisions and live in the app's Settings → Integrations, where every connected repository is visible at once.

## Request body

- EvalCheckRepoConnectRequest
  - `projectId` string, required
  - `suiteId` string, required — The eval suite this repository's pull requests will run.
  - `repo` string, required — `owner/repo`. Canonicalized and VERIFIED against the App's installation before it is stored.
  - `outagePolicy` 'fail_open' | 'fail_closed', required — REQUIRED here, though the platform leaves it optional for a deploy window: a surface that defaults it is the one that quietly produces repositories nobody chose a policy for.

## Response `201`

Connected.

- EvalCheckRepoConnected
  - `id` string, required
  - `organizationId` string, required
  - `projectId` string, required
  - `suiteId` string, required
  - `repo` string, required
  - `outagePolicy` 'fail_open' | 'fail_closed', required

## Other responses

- `400` — Malformed body or parameters.
- `401` — Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key.
- `403` — Key is valid but not allowed to do this.
- `404` — Unknown project, server, or resource.
- `429` — Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter.
- `500` — Something failed on MCPJam's side.

---

[API](https://skmtc.net/mcpjam/apis/mcpjam-api.md) · [All operations](https://skmtc.net/mcpjam/apis/mcpjam-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mcpjam/mcpjam-api/revisions/8978a4e735fa/schema)