Sign in

v1

latestOpenAPI 3.0.02026-07-1346146138.4 KB
evaluators

Execute agent-tool-trajectory evaluator

Compare actual tool calls against expected reference tool calls

Request Body:

  • input.executed_tool_calls (string, required): JSON array of actual tool calls made by the agent
  • input.expected_tool_calls (string, required): JSON array of expected/reference tool calls
  • config.threshold (float, optional): Score threshold for pass/fail determination (default: 0.5)
  • config.mismatch_sensitive (bool, optional): Whether tool calls must match exactly (default: false)
  • config.order_sensitive (bool, optional): Whether order of tool calls matters (default: false)
  • config.input_params_sensitive (bool, optional): Whether to compare input parameters (default: true)
post/v2/evaluators/agent-tool-trajectory/execute

Request body

Example request

{
  "config": {
    "input_params_sensitive": true,
    "threshold": 0.5
  },
  "input": {
    "executed_tool_calls": "[{\"name\": \"search\", \"input\": {\"query\": \"weather\"}}]",
    "expected_tool_calls": "[{\"name\": \"search\", \"input\": {\"query\": \"weather\"}}]"
  }
}

Response

OK

reasonstring
scorenumber
successboolean

Example response

{
  "reason": "Tool calls match the expected trajectory",
  "score": 0.85,
  "success": true
}