Evals
Run an eval suite on a repository's pull requests
Connect a repository so every pull request to it runs one eval suite and reports a GitHub check. Affects everyone who opens a pull request on that repository, and with outagePolicy: fail_closed it can block their merges.
The repository is verified against the organization's GitHub App installation before it is stored. A repository that does not exist and one the App cannot see answer identically, on purpose: answering differently would make this an oracle for private repository names.
Deliberately narrow: retargeting a repository at a different suite, pausing it, and disconnecting are repo-level decisions and live in the app's Settings → Integrations, where every connected repository is visible at once.
post/organizations/{organizationId}/eval-check-repos
Request body
Response
Connected.