v1

latestOpenAPI 3.1.02026-07-22152755.8 KB

test_framework

Retrieve an evaluator (scenario)

Fetch a single evaluator by id. Returns full configuration including instructions, expected outcome, metrics, test profile, and — when the agent has mock tools attached and the scenario was auto-generated — the expected mock tool calls (generated_mock_tool_entries) with their inputs and outputs.

get/test_framework/v1/scenarios/{id}/

Path parameters

idinteger required

A unique integer value identifying this scenario.

Response

idinteger

namestring

Name of the evaluator

agentinteger

Agent ID Example: 2142

projectinteger nullable

assistant_idstring

Alternative to agent ID - the assistant ID to use for this scenario Example: "asst_1234567890"

personalityinteger

ID of the personality

personality_namestring

Name of the personality

agent_namestring

Name of the agent associated with this scenario Example: "Customer Support Agent"

tagsstring[]

List of tags of the evaluators Example: ["tag1", "tag2", "tag3"]

{"stackTrail":"components:schemas:SchemaScenario:properties:tool_ids","oasType":"schema","type":"unknown","description":"\nList of tool IDs to associate with this scenario\nExample: `[\"TOOL_DTMF\", \"TOOL_END_CALL\"]`\n"}

metricsinteger[]

List of metric IDs Example: [123, 456, 789]

metric_namesstring[]

List of metric names Example: ["Metric 1", "Metric 2", "Metric 3"]

phone_numberstring

Phone number eg: +17776666333

{"stackTrail":"components:schemas:PhoneNumberInbound","oasType":"schema","type":"unknown","description":"Referenced by this API but not published in the vendor's docs pages."}

first_messagestring

First message of the evaluator

inbound_phone_numberinteger nullable

{"stackTrail":"components:schemas:PhoneNumberInbound","oasType":"schema","type":"unknown","description":"Referenced by this API but not published in the vendor's docs pages."}

instructionsstring

Scenario Instructions - format depends on scenario_type:

For instruction type: Plain text instructions
For conditional_actions type: JSON string with role and conditions array. Each condition has id, condition trigger (FIRST_MESSAGE, a natural-language third-person description of the main agent's observable action — e.g. "The agent asks for your date of birth", matched semantically — or an integer reference to another condition's id), action (text with optional <silence time="1.5s" /> (interruptible) or <hold time="2s" /> (not interruptible) and [laughter] tags), type (standard/action_followup), and fixed_message flag

{"stackTrail":"components:schemas:SchemaScenario:properties:conditional_actions","oasType":"schema","type":"unknown","description":"Structured conditional-actions object for `scenario_type=\"conditional_actions\"`. Use either `conditional_actions` (structured) or `instructions` (plain text) — not both.\n\nExample:"}