---
title: "Invoke a Custom Evaluator"
method: POST
path: "/v2/evaluators/{id}/invoke"
tags: ["Evals"]
---

# Invoke a Custom Evaluator

`POST /v2/evaluators/{id}/invoke`

## Path parameters

- `id` string, required — Evaluator ID

## Request body

- object
  - `query` string — Latest user message
  - `output` string — The generated response from the model
  - `reference` string — The reference used to compare the output
  - `retrievals` string[] — Knowledge base retrievals
  - `messages` object[] — The messages used to generate the output, without the last user message
    - `role` 'system' | 'developer' | 'assistant' | 'user' | 'exception' | 'tool' | 'prompt' | 'correction' | 'expected_output', required — The role of the prompt message
    - `content` union, required — The contents of the user message. Either the text content of the message or an array of content parts with a defined type, each can be of type `text` or `image_url` when passing in images. You can pass multiple images by adding multiple `image_url` content parts. Can be null for tool messages in certain scenarios.
      - string
      - union[]
        - union
          - object — Text content part of a prompt message
            - `type` 'text', required
            - `text` string, required
          - object — The image part of the prompt message. Only supported with vision models.
            - `type` 'image_url', required
            - `image_url` object, required
              - …
          - object
            - `type` 'file', required — The type of the content part. Always `file`.
            - `file` object, required
              - …
    - `tool_calls` object[]
      - `id` string
      - `index` number
      - `type` 'function', required
      - `function` object, required
        - `name` string, required
        - `arguments` string, required — JSON string arguments for the functions
    - `tool_call_id` string, nullable
  - `model` string — Model to use for LLM-based evaluators (e.g. "openai/gpt-4o")
  - `variables` EvaluatorVariables — Template variables for evaluator prompt substitution. Request values override evaluator defaults, including for nested arrays and objects.

## Response `200`

Returns the result of the evaluator run

- union
  - object
    - `type` 'string', required
    - `original_value` string, nullable
    - `value` string, nullable
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'number', required
    - `original_value` number, nullable
    - `value` number, nullable, required
    - `format_options` union
      - object
        - `significant_digits` number, required
      - object
        - `style` 'currency', required
        - `currency` 'USD', required
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'boolean', required
    - `value` union, required
      - boolean
      - string
      - number
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'string_array', required
    - `values` string[], required
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'rouge_n', required
    - `value` object, required
      - `rouge_1` object, required
        - `f1` number, required
        - `precision` number, required
        - `recall` number, required
      - `rouge_2` object, required
        - `f1` number, required
        - `precision` number, required
        - `recall` number, required
      - `rouge_l` object, required
        - `f1` number, required
        - `precision` number, required
        - `recall` number, required
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'bert_score', required
    - `value` object, required
      - `f1` number, required
      - `precision` number, required
      - `recall` number, required
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'llm_evaluator', required
    - `value` object, nullable, required
      - `workflow_run_id` string, nullable
      - `trace_id` string
      - `span_id` string
      - `value` union, required
        - number
        - boolean
        - string
      - `explanation` string, nullable
      - `original_value` union
        - number
        - boolean
        - string
      - `original_explanation` string, nullable
      - `jury` object
        - `judges_configured` integer, required
        - `judges_succeeded` integer, required
        - `judges_failed` integer, required
        - `replacements_used` integer, required
        - `tie` boolean, required
        - `votes` object[], required
          - `model` string, required
          - `replacement` boolean, required
          - `success` boolean, required
          - `value` union
            - number
            - boolean
            - string
          - `explanation` string
          - `error` string
        - `stats` object
          - `mean` number, required
          - `std` number, required
  - object
    - `type` 'http_eval', required
    - `value` object, nullable
      - `type` string, required
      - `value` union, required
        - number
        - boolean
    - `trace_id` string
    - `span_id` string
  - object
    - `type` 'structured', required
    - `value` object, required
    - `trace_id` string
    - `span_id` string

## Other responses

- `400` — Bad request
- `404` — Workspace ID is not found on the request
- `500` — Error running the evaluator

---

[API](https://skmtc.net/orq-ai/apis/orq-ai-api.md) · [All operations](https://skmtc.net/orq-ai/apis/orq-ai-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/orq-ai/orq-ai-api/versions/41c02aedc7c0/schema)