Sign in

v1

latestOpenAPI 3.0.02026-07-1346146138.4 KB
evaluators

Execute agent-tool-selection-quality evaluator

Evaluate whether the agent selected the most appropriate tool(s) for the task at hand

Request Body:

  • input.input_messages (array of objects, required): The agent's input messages — system prompt + conversation history as a JSON array of message objects (role, content, optional tool_calls, etc.)
  • input.output_messages (array of objects, required): The agent's output messages for this step as a JSON array of message objects (may contain tool calls or plain text)
post/v2/evaluators/agent-tool-selection-quality/execute

Request body

Example request

{
  "input": {
    "input_messages": "[{\"role\":\"system\",\"content\":\"You are a helpful assistant with access to tools: 1. get_weather(city) - Get current weather. 2. search_web(query) - Search the web.\"},{\"role\":\"user\",\"content\":\"What is the weather like in Paris?\"}]",
    "output_messages": "[{\"role\":\"assistant\",\"tool_calls\":[{\"name\":\"get_weather\",\"arguments\":{\"city\":\"Paris\"}}]}]"
  }
}

Response

OK

correct_argsboolean
correct_args_reasonstring
tool_selectionstring
tool_selection_reasonstring

Example response

{
  "correct_args": true,
  "correct_args_reason": "The city argument matches the user's request",
  "tool_selection": "correct",
  "tool_selection_reason": "The agent correctly chose get_weather for the weather query"
}