Create a Scenario.

Create a Scenario, a repeatable AI coding evaluation test that defines the starting environment as well as evaluation success criteria.

Request body

namestring required

Name of the scenario.

metadataobject nullable

User defined metadata to attach to the scenario for organization.

reference_outputstring nullable

A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.

required_environment_variablesstring[] nullable

Environment variables required to run the scenario. If these variables are not provided, the scenario will fail to start.

required_secret_namesstring[] nullable

Secrets required to run the scenario (user secret name to scenario required secret name). If these secrets are not provided or the mapping is incorrect, the scenario will fail to start.

validation_type'UNSPECIFIED' | 'FORWARD' | 'REVERSE' | 'EVALUATION'

scorer_timeout_secinteger nullable

Timeout for scoring in seconds. Default 30 minutes (1800s).

Response

OK

idstring required

The ID of the Scenario.

namestring required

The name of the Scenario.

metadataobject required

User defined metadata to attach to the scenario for organization.

reference_outputstring nullable

A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.

required_environment_variablesstring[]

Environment variables required to run the scenario. If any required environment variables are missing, the scenario will fail to start.

required_secret_namesstring[]

Environment variables required to run the scenario. If any required secrets are missing, the scenario will fail to start.

is_publicboolean

Whether this scenario is public.