Sign in

v1

latestOpenAPI 3.1.02026-07-26142186376.1 KB
Evals

Execute Evaluation

Run evaluation tests on agents or teams. Supports accuracy, agent-as-judge, performance, and reliability evaluations. Requires either agent_id or team_id, but not both.

post/eval-runs

Query parameters

db_idstring nullable

Database ID to use for evaluation

Database ID to use for evaluation

tablestring nullable

Table to use for evaluation

Table to use for evaluation

Request body

agent_idstring nullable

Agent ID to evaluate

team_idstring nullable

Team ID to evaluate

model_idstring nullable

Model ID to use for evaluation

model_providerstring nullable

Model provider name

eval_type'accuracy' | 'agent_as_judge' | 'performance' | 'reliability' required
inputstring required

Input text/query for the evaluation

additional_guidelinesstring nullable

Additional guidelines for the evaluation

additional_contextstring nullable

Additional context for the evaluation

num_iterationsinteger

Number of times to run the evaluation

namestring nullable

Name for this evaluation run

expected_outputstring nullable

Expected output for accuracy evaluation

criteriastring nullable

Evaluation criteria for agent-as-judge evaluation

scoring_strategy'numeric' | 'binary' nullable

Scoring strategy: 'numeric' (1-10 with threshold) or 'binary' (PASS/FAIL)

thresholdinteger nullable

Score threshold for pass/fail (1-10), only used with numeric scoring

warmup_runsinteger

Number of warmup runs before measuring performance

expected_tool_callsstring[] nullable

Expected tool calls for reliability evaluation

allow_additional_tool_callsboolean

When True, tool calls not in expected_tool_calls are allowed (subset matching)

expected_tool_call_argumentsobject nullable

Expected arguments for specific tool calls, e.g. {"tool_name": {"arg_name": "expected_value"}} or {"tool_name": [{"arg_name": "val1"}, {"arg_name": "val2"}]}

Response

Evaluation executed successfully

idstring required

Unique identifier for the evaluation run

agent_idstring nullable

Agent ID that was evaluated

model_idstring nullable

Model ID used in evaluation

model_providerstring nullable

Model provider name

team_idstring nullable

Team ID that was evaluated

workflow_idstring nullable

Workflow ID that was evaluated

namestring nullable

Name of the evaluation run

evaluated_component_namestring nullable

Name of the evaluated component

eval_type'accuracy' | 'agent_as_judge' | 'performance' | 'reliability' required
eval_dataobject required

Evaluation results and metrics

eval_inputobject nullable

Input parameters used for the evaluation

created_atstring date-time nullable

Timestamp when evaluation was created

updated_atstring date-time nullable

Timestamp when evaluation was last updated