latestOpenAPI 3.1.0raw.githubusercontent.com2026-08-21161224637.5 KB

8978a4e735fa

Evals

Run an eval suite on a repository's pull requests

Connect a repository so every pull request to it runs one eval suite and reports a GitHub check. Affects everyone who opens a pull request on that repository, and with outagePolicy: fail_closed it can block their merges.

The repository is verified against the organization's GitHub App installation before it is stored. A repository that does not exist and one the App cannot see answer identically, on purpose: answering differently would make this an oracle for private repository names.

Deliberately narrow: retargeting a repository at a different suite, pausing it, and disconnecting are repo-level decisions and live in the app's Settings → Integrations, where every connected repository is visible at once.

post/organizations/{organizationId}/eval-check-repos

Request body

projectIdstring required
suiteIdstring required

The eval suite this repository's pull requests will run.

repostring required

owner/repo. Canonicalized and VERIFIED against the App's installation before it is stored.

outagePolicy'fail_open' | 'fail_closed' required

REQUIRED here, though the platform leaves it optional for a deploy window: a surface that defaults it is the one that quietly produces repositories nobody chose a policy for.

Response

Connected.

idstring required
organizationIdstring required
projectIdstring required
suiteIdstring required
repostring required
outagePolicy'fail_open' | 'fail_closed' required