---
title: "Invoke a Custom Evaluator"
method: POST
path: "/v3/evaluators/{id}/invoke"
tags: ["Evals"]
---

# Invoke a Custom Evaluator

`POST /v3/evaluators/{id}/invoke`

Runs an evaluator that already exists in the workspace. Accepts either a conversation or the structured input and output fields; when both are present the conversation wins.

## Path parameters

- `id` string, required

## Request body

- InvokeEvaluatorRequest — Accepts two shapes. `context` names its fields after the template variables they feed and is the one to use; the flat fields below are the legacy body, folded into `context` when it is absent. Setting `context` wins.
  - `context` EvaluationContext — The data to grade. When `messages` is present it is the conversation and `input.user_query` is ignored; `output.response` is appended only when the conversation carries no assistant turn. Mirrors graders-api buildGraderRequest.
    - `messages` object[]
    - `input` StructuredInput — StructuredInput names its fields after the template variables they feed, so input.user_query in a prompt is user_query here.
      - `system_instructions` string
      - `user_query` string
      - `retrievals` string[]
      - `expected_output` string
    - `output` StructuredOutput
      - `response` string
      - `tools_called` StructuredToolCall[]
        - `name` string
        - `arguments` string
        - `output` string
    - `variables` object
  - `model` string — Model to grade with, as a catalog id such as "openai/gpt-4o". Only meaningful for a hub template of type llm_eval or ragas, which has no model of its own. A stored evaluator uses the model on its own definition and ignores this.
  - `query` string — Latest user message. Folds into `context.input.user_query`.
  - `output` string — The generated response from the model. Folds into `context.output.response`.
  - `reference` string — The reference used to compare the output. Folds into `context.input.expected_output`.
  - `retrievals` string[] — Knowledge base retrievals. Folds into `context.input.retrievals`.
  - `messages` object[] — The conversation that produced the output. Folds into `context.messages`.
  - `variables` object — Template variables for evaluator prompt substitution. Folds into `context.variables`.

## Response `200`

OK

- InvokeEvaluatorResponse — Response wrappers keep each RPC's response type distinct, so a future field can be added to one without touching the other.
  - `result` EvaluationResult — The verdict. Shaped to match WorkflowRunMinifiedEvalSchema, the body the TypeScript route returned, so existing consumers read the same JSON.
    - `type` string — Discriminator, matching the legacy union: "string", "number", "boolean", "string_array", "rouge_n", "bert_score", "llm_evaluator", "http_eval".
    - `value` unknown
    - `trace_id` string — Trace reference of the evaluator's own span. Optional so an absent reference is omitted rather than emitted as an empty string, matching the legacy body.
    - `span_id` string
    - `evaluator_id` string
    - `status` string — How the run ended, as distinct from `passed`: "passed", "condition_failed" or "timed_out". A string, not an enum, because the engine owns the vocabulary.
    - `passed` boolean — The guardrail's decision when the evaluator has one, the grader's own judgement otherwise. Always present — the endpoint this replaces omitted it without a guardrail, so read `guardrail_config` to detect one, not this.
    - `explanation` string
    - `categories` string[] — Set by the classifying graders (moderation, PII, secret detection), which report which categories tripped rather than a single verdict.
    - `confidence` number, double — Optional so an absent score is omitted rather than reported as 0.0, which a consumer would read as maximum uncertainty.

---

[API](https://skmtc.net/orq-ai/apis/orq-ai-api.md) · [All operations](https://skmtc.net/orq-ai/apis/orq-ai-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/orq-ai/orq-ai-api/revisions/2cd7e7f7bcb4/schema)