---
title: "Create evaluator"
method: POST
path: "/v2/evaluators"
tags: ["Evaluators"]
---

# Create evaluator

`POST /v2/evaluators`

Creates a new evaluator with an initial version.

**Payload Requirements**
- The evaluator `name` must be unique within the given space.
- `type` (top-level) selects the evaluator kind: `TEMPLATE` or `CODE`.
  With `TEMPLATE`, provide `version.template_config`.
  With `CODE`, provide `version.code_config` — where `code_config.type` is `MANAGED` or `CUSTOM` (a separate discriminator *within* `code_config`, independent of the top-level `type: CODE`).
- For template evaluators: `version.template_config.name` is the eval column name; must match `^[a-zA-Z0-9_\s\-&()]+$`.
- For template evaluators: `version.template_config.template` is the prompt template; use `{variable}` for placeholders (f-string format, e.g. `{input}`, `{output}`).
- For template evaluators: `version.template_config.classification_choices` is required and maps choice labels to numeric scores (e.g. `{"relevant": 1, "irrelevant": 0}`).
- For code evaluators: see `CodeConfig` — managed evaluators (`code_config.type: MANAGED`) use `managed_evaluator` and `variables`; custom evaluators (`code_config.type: CUSTOM`) use `code`, optional `imports`, and `variables`.
- System-managed fields (`id`, `created_at`, `updated_at`, `created_by_user_id`) are rejected on input.

**Valid example** (template evaluator)
```json
{
  "name": "Hallucination Detector",
  "space_id": "U3BhY2U6MTpWNEth",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Initial version",
    "template_config": {
      "name": "hallucination",
      "template": "Given the input: {input}\nand the output: {output}\nIs the output a hallucination?",
      "include_explanations": true,
      "use_function_calling_if_available": true,
      "classification_choices": {"hallucinated": 0, "factual": 1},
      "llm_config": {
        "ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
        "model_name": "gpt-4o",
        "invocation_parameters": {"temperature": 0},
        "provider_parameters": {}
      }
    }
  }
}
```

**Invalid example** (type/config mismatch — `TEMPLATE` type with `code_config`)
```json
{
  "name": "Bad Evaluator",
  "space_id": "U3BhY2U6MTpWNEth",
  "type": "TEMPLATE",
  "version": {
    "commit_message": "Wrong config",
    "code_config": {
      "type": "CUSTOM",
      "name": "my_eval",
      "code": "class Evaluator: ...",
      "variables": ["input"]
    }
  }
}
```

<Note>This endpoint is in beta, read more [here](https://arize.com/docs/ax/rest-reference#api-version-stages).</Note>

## Request body

- CreateEvaluatorRequest — Body containing evaluator creation parameters with an initial version. Only `type: TEMPLATE` and `type: CODE` are currently accepted on creation.
  - `space_id` string, required — Space identifier (base64)
  - `name` string, required — Evaluator name (must be unique within the space)
  - `description` string — Evaluator description
  - `type` 'TEMPLATE' | 'CODE' | 'HARNESS' | 'REMOTE', required — The evaluator type: - `TEMPLATE` — LLM-based evaluator. - `CODE` — managed built-in evaluators or custom Python code (both are subtypes of `CODE`, discriminated by the nested `CodeConfig.type` = `MANAGED` | `CUSTOM`). - `HARNESS` — test harness evaluator. - `REMOTE` — remote evaluator. Applies to both the parent `Evaluator.type` field and every version's `type` discriminator — a version's `type` must always match its parent evaluator's `type`.
  - `version` union, required — Payload for an evaluator version: exactly one of `template_config` or `code_config`. Used both when creating an evaluator (initial `version`) and when appending a version.
    - CreateTemplateEvaluatorVersionRequest
      - `commit_message` string, required — Commit message describing the changes
      - `template_config` TemplateConfigInput, required
        - `name` string, required — Eval column name. Must match ^[a-zA-Z0-9_\s\-&()]+$
        - `template` string, required — The prompt template with variable placeholders
        - `include_explanations` boolean, required — Whether to include explanations in the evaluation output
        - `use_function_calling_if_available` boolean, required — Whether to use function calling if the model supports it
        - `use_structured_output` boolean — Whether to use structured output if the model supports it. When omitted the server defaults to true.
        - `classification_choices` object, nullable — Map of choice label to numeric score (e.g. {"relevant": 1, "irrelevant": 0}). When omitted, the evaluator produces freeform (non-classification) output.
        - `direction` 'MAXIMIZE' | 'MINIMIZE' | 'NONE' — The direction for optimization. Defaults to `NONE` when omitted. - MAXIMIZE: higher scores are better - MINIMIZE: lower scores are better - NONE: higher or lower scores are neither better nor worse
        - `data_granularity` 'SPAN' | 'TRACE' | 'SESSION' — Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.
        - `llm_config` EvaluatorLlmConfigRequest, required — LLM configuration for an evaluator in a write request (strict form of EvaluatorLlmConfig)
          - `ai_integration_id` string, required — AI integration identifier (base64)
          - `model_name` string, required — Model name (e.g. gpt-4o)
          - `invocation_parameters` InvocationParamsRequest, required — Parameters for the LLM invocation in a write request (strict form of InvocationParams; leaf schemas use *Request variants)
            - `temperature` number — Sampling temperature (higher = more random)
            - `max_tokens` integer — Maximum number of tokens to generate
            - `max_completion_tokens` integer — Maximum number of completion tokens to generate
            - `top_p` number — Nucleus sampling parameter
            - `frequency_penalty` number — Frequency penalty (-2.0 to 2.0)
            - `presence_penalty` number — Presence penalty (-2.0 to 2.0)
            - `stop` string[] — Stop sequences
            - `response_format` ResponseFormatRequest — Response format configuration in a write request (strict form of ResponseFormat)
              - …
            - `tool_config` ToolConfig — Tool configuration for the LLM invocation
              - …
            - `top_k` integer — Top-K sampling parameter. A top-K of 1 means the next selected token is the most probable (greedy decoding).
            - `thinking_level` string — Controls how much reasoning the model performs before responding. Supported by Gemini 3.x models. Accepted values: 'low', 'high'.
            - `thinking_budget` integer — Maximum tokens the model may use for internal reasoning. Supported by Gemini 2.5 models. Range: 0-24576 (Flash/Flash-Lite) or 128-32768 (Pro). Set 0 to disable thinking on Flash models.
            - `reasoning_effort` string — Controls how much reasoning the model performs before responding. Supported by OpenAI o-series and GPT-5 models. o-series: 'low' | 'medium' | 'high'. GPT-5: 'none' | 'low' | 'medium' | 'high' | 'xhigh'.
            - `verbosity` string — Controls the verbosity of model output. Supported by OpenAI GPT-5 series. Accepted values: 'low' | 'medium' | 'high'.
          - `provider_parameters` ProviderParamsRequest, required — Provider-specific parameters in a write request (strict form of ProviderParams; leaf schemas use *Request variants)
            - `azure_params` AzureParamsRequest — Azure OpenAI specific parameters in a write request (strict form of AzureParams)
              - …
            - `anthropic_headers` AnthropicHeadersRequest — Anthropic-specific headers in a write request (strict form of AnthropicHeaders)
              - …
            - `anthropic_version` string — Anthropic API version
            - `bedrock_options` BedrockOptionsRequest — AWS Bedrock options in a write request (strict form of BedrockOptions)
              - …
            - `region` string — Region for the model deployment
    - CreateCodeEvaluatorVersionRequest
      - `commit_message` string, required — Commit message describing the changes
      - `code_config` union, required — Strict request form of CodeConfig. Discriminated union of ManagedCodeConfigRequest and CustomCodeConfigRequest. Use in write request bodies.
        - ManagedCodeConfigRequest — Managed (built-in) code evaluator configuration in a write request (strict form of ManagedCodeConfig)
          - `data_granularity` 'SPAN' | 'TRACE' | 'SESSION' — Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.
          - `query_filter` string, nullable — Optional filter query over the chosen data granularity. When omitted or null, no filter is applied.
          - `type` 'MANAGED', required — Discriminator identifying this as a managed (built-in) code evaluator
          - `name` string, required — Eval column name. Must match ^[a-zA-Z0-9_\s\-&()]+$
          - `managed_evaluator` 'MATCHES_REGEX' | 'JSON_PARSEABLE' | 'CONTAINS_ANY_KEYWORD' | 'CONTAINS_ALL_KEYWORDS' | 'EXACT_MATCH', required — Built-in managed code evaluator name
          - `variables` string[], required — Dataset columns or span attributes passed into the evaluator (order and count must match the managed evaluator's requirements).
          - `static_params` StaticParamRequest[] — Static parameters for the managed evaluator (see registry `args`). When omitted, the registry's required arguments must be satisfied by defaults on the evaluator class; otherwise validation fails with 400. If the registry has no args, omitting this field is equivalent to an empty list.
            - `name` string, required — Parameter name (matches the managed evaluator's argument name)
            - `type` 'STRING' | 'STRING_ARRAY' | 'REGEX', required — Argument type for static evaluator parameters. - STRING - A single string value. - STRING_ARRAY - An array of string values. - REGEX - A regular expression string.
            - `default_value` union, required — Default value. Must be a string when `type` is STRING or REGEX, and a string array when `type` is STRING_ARRAY. Mismatches are rejected with 400 by the server.
              - …
        - CustomCodeConfigRequest — Custom (user-supplied Python) code evaluator configuration in a write request (strict form of CustomCodeConfig)
          - `data_granularity` 'SPAN' | 'TRACE' | 'SESSION' — Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.
          - `query_filter` string, nullable — Optional filter query over the chosen data granularity. When omitted or null, no filter is applied.
          - `type` 'CUSTOM', required — Discriminator identifying this as a custom (user-supplied Python) code evaluator
          - `name` string, required — Eval column name. Must match ^[a-zA-Z0-9_\s\-&()]+$
          - `code` string, required — Python source defining the evaluator class
          - `imports` string, nullable — Optional package import block prepended when running the evaluator
          - `variables` string[], required — Dataset columns or span attributes mapped to evaluate() arguments
          - `static_params` StaticParamRequest[] — Optional typed defaults accessible on the evaluator instance. Omit or pass an empty array when the custom class does not read any static parameters.
            - `name` string, required — Parameter name (matches the managed evaluator's argument name)
            - `type` 'STRING' | 'STRING_ARRAY' | 'REGEX', required — Argument type for static evaluator parameters. - STRING - A single string value. - STRING_ARRAY - An array of string values. - REGEX - A regular expression string.
            - `default_value` union, required — Default value. Must be a string when `type` is STRING or REGEX, and a string array when `type` is STRING_ARRAY. Mismatches are rejected with 400 by the server.
              - …

## Response `201`

Returns an evaluator with a resolved version

- EvaluatorWithVersion — An evaluator defines reusable evaluation logic that can be attached to evaluation tasks. The type field determines the kind of evaluation: TEMPLATE (LLM-based template evaluation) or CODE (custom code evaluation).
  - `id` string, required — The unique identifier for the evaluator
  - `name` string, required — The name of the evaluator
  - `description` string, nullable — The description of the evaluator
  - `type` 'TEMPLATE' | 'CODE' | 'HARNESS' | 'REMOTE', required — The evaluator type: - `TEMPLATE` — LLM-based evaluator. - `CODE` — managed built-in evaluators or custom Python code (both are subtypes of `CODE`, discriminated by the nested `CodeConfig.type` = `MANAGED` | `CUSTOM`). - `HARNESS` — test harness evaluator. - `REMOTE` — remote evaluator. Applies to both the parent `Evaluator.type` field and every version's `type` discriminator — a version's `type` must always match its parent evaluator's `type`.
  - `space_id` string, required — The unique identifier for the space the evaluator belongs to
  - `created_at` string, date-time, required — When the evaluator was created
  - `updated_at` string, date-time, required — When the evaluator was last updated
  - `created_by_user_id` string, nullable, required — The unique identifier for the user who created the evaluator
  - `version` union, required — A versioned snapshot of an evaluator's configuration. The `type` field discriminates the branch and matches the parent evaluator's `type`.
    - EvaluatorVersionTemplate — Evaluator version carrying a template (LLM) configuration.
      - `id` string, required — The unique identifier for this version
      - `evaluator_id` string, required — The parent evaluator ID
      - `commit_hash` string, required — A unique hash identifying this version
      - `commit_message` string, nullable, required — A message describing the changes in this version
      - `created_at` string, date-time, required — When this version was created
      - `created_by_user_id` string, nullable, required — The unique identifier for the user who created this version
      - `type` 'TEMPLATE', required — Discriminator identifying this as a template evaluator version. Always `TEMPLATE` for this variant.
      - `template_config` TemplateConfig, required
        - `name` string, required — Eval column name. Must match ^[a-zA-Z0-9_\s\-&()]+$
        - `template` string, required — The prompt template with variable placeholders
        - `include_explanations` boolean, required — Whether to include explanations in the evaluation output
        - `use_function_calling_if_available` boolean, required — Whether to use function calling if the model supports it
        - `use_structured_output` boolean — Whether to use structured output if the model supports it
        - `classification_choices` object, nullable — Map of choice label to numeric score (e.g. {"relevant": 1, "irrelevant": 0}). Null for legacy freeform evaluators that predate required choices.
        - `direction` 'MAXIMIZE' | 'MINIMIZE' | 'NONE' — The direction for optimization. Defaults to `NONE` when omitted. - MAXIMIZE: higher scores are better - MINIMIZE: lower scores are better - NONE: higher or lower scores are neither better nor worse
        - `data_granularity` 'SPAN' | 'TRACE' | 'SESSION' — Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.
        - `llm_config` EvaluatorLlmConfig, required
          - `ai_integration_id` string, required — AI integration identifier (base64)
          - `model_name` string, required — Model name (e.g. gpt-4o)
          - `invocation_parameters` InvocationParams, required — Parameters for the LLM invocation
            - `temperature` number — Sampling temperature (higher = more random)
            - `max_tokens` integer — Maximum number of tokens to generate
            - `max_completion_tokens` integer — Maximum number of completion tokens to generate
            - `top_p` number — Nucleus sampling parameter
            - `frequency_penalty` number — Frequency penalty (-2.0 to 2.0)
            - `presence_penalty` number — Presence penalty (-2.0 to 2.0)
            - `stop` string[] — Stop sequences
            - `response_format` ResponseFormat — Response format configuration
              - …
            - `tool_config` ToolConfig — Tool configuration for the LLM invocation
              - …
            - `top_k` integer — Top-K sampling parameter. A top-K of 1 means the next selected token is the most probable (greedy decoding).
            - `thinking_level` string — Controls how much reasoning the model performs before responding. Supported by Gemini 3.x models. Accepted values: 'low', 'high'.
            - `thinking_budget` integer — Maximum tokens the model may use for internal reasoning. Supported by Gemini 2.5 models. Range: 0-24576 (Flash/Flash-Lite) or 128-32768 (Pro). Set 0 to disable thinking on Flash models.
            - `reasoning_effort` string — Controls how much reasoning the model performs before responding. Supported by OpenAI o-series and GPT-5 models. o-series: 'low' | 'medium' | 'high'. GPT-5: 'none' | 'low' | 'medium' | 'high' | 'xhigh'.
            - `verbosity` string — Controls the verbosity of model output. Supported by OpenAI GPT-5 series. Accepted values: 'low' | 'medium' | 'high'.
          - `provider_parameters` ProviderParams, required — Provider-specific parameters
            - `azure_params` AzureParams — Azure OpenAI specific parameters
              - …
            - `anthropic_headers` AnthropicHeaders — Anthropic-specific headers
              - …
            - `anthropic_version` string — Anthropic API version
            - `bedrock_options` BedrockOptions — AWS Bedrock options
              - …
            - `region` string — Region for the model deployment
    - EvaluatorVersionCode — Evaluator version carrying a code configuration.
      - `id` string, required — The unique identifier for this version
      - `evaluator_id` string, required — The parent evaluator ID
      - `commit_hash` string, required — A unique hash identifying this version
      - `commit_message` string, nullable, required — A message describing the changes in this version
      - `created_at` string, date-time, required — When this version was created
      - `created_by_user_id` string, nullable, required — The unique identifier for the user who created this version
      - `type` 'CODE', required — Discriminator identifying this as a code evaluator version. Always `CODE` for this variant.
      - `code_config` union, required — Discriminated union representing either a managed (built-in) or custom (user-supplied Python) code evaluator configuration, resolved by the nested `type` field (`MANAGED` -> `ManagedCodeConfig`, `CUSTOM` -> `CustomCodeConfig`). This inner `type` is independent of the parent evaluator version's `type` (which is always `CODE` here).
        - ManagedCodeConfig
          - `data_granularity` 'SPAN' | 'TRACE' | 'SESSION' — Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.
          - `query_filter` string, nullable — Optional filter query over the chosen data granularity. When omitted or null, no filter is applied.
          - `type` 'MANAGED', required — Discriminator identifying this as a managed (built-in) code evaluator
          - `name` string, required — Eval column name. Must match ^[a-zA-Z0-9_\s\-&()]+$
          - `managed_evaluator` 'MATCHES_REGEX' | 'JSON_PARSEABLE' | 'CONTAINS_ANY_KEYWORD' | 'CONTAINS_ALL_KEYWORDS' | 'EXACT_MATCH', required — Built-in managed code evaluator name
          - `variables` string[], required — Dataset columns or span attributes passed into the evaluator (order and count must match the managed evaluator's requirements).
          - `static_params` StaticParam[] — Static parameters for the managed evaluator (see registry `args`). When omitted, the registry's required arguments must be satisfied by defaults on the evaluator class; otherwise validation fails with 400. If the registry has no args, omitting this field is equivalent to an empty list.
            - `name` string, required — Parameter name (matches the managed evaluator's argument name)
            - `type` 'STRING' | 'STRING_ARRAY' | 'REGEX', required — Argument type for static evaluator parameters. - STRING - A single string value. - STRING_ARRAY - An array of string values. - REGEX - A regular expression string.
            - `default_value` union, required — Default value. Must be a string when `type` is STRING or REGEX, and a string array when `type` is STRING_ARRAY. Mismatches are rejected with 400 by the server.
              - …
        - CustomCodeConfig
          - `data_granularity` 'SPAN' | 'TRACE' | 'SESSION' — Data granularity level for evaluation. - SPAN - Evaluate at the individual span level. - TRACE - Evaluate at the full trace level. - SESSION - Evaluate at the session level.
          - `query_filter` string, nullable — Optional filter query over the chosen data granularity. When omitted or null, no filter is applied.
          - `type` 'CUSTOM', required — Discriminator identifying this as a custom (user-supplied Python) code evaluator
          - `name` string, required — Eval column name. Must match ^[a-zA-Z0-9_\s\-&()]+$
          - `code` string, required — Python source defining the evaluator class
          - `imports` string, nullable — Optional package import block prepended when running the evaluator
          - `variables` string[], required — Dataset columns or span attributes mapped to evaluate() arguments
          - `static_params` StaticParam[] — Optional typed defaults accessible on the evaluator instance. Omit or pass an empty array when the custom class does not read any static parameters.
            - `name` string, required — Parameter name (matches the managed evaluator's argument name)
            - `type` 'STRING' | 'STRING_ARRAY' | 'REGEX', required — Argument type for static evaluator parameters. - STRING - A single string value. - STRING_ARRAY - An array of string values. - REGEX - A regular expression string.
            - `default_value` union, required — Default value. Must be a string when `type` is STRING or REGEX, and a string array when `type` is STRING_ARRAY. Mismatches are rejected with 400 by the server.
              - …
    - EvaluatorVersionHarness — Evaluator version backed by a harness evaluation config. Only common version metadata (id, commit info, timestamps) is returned — the harness configuration is not yet accessible and will be a future addition.
      - `id` string, required — The unique identifier for this version
      - `evaluator_id` string, required — The parent evaluator ID
      - `commit_hash` string, required — A unique hash identifying this version
      - `commit_message` string, nullable, required — A message describing the changes in this version
      - `created_at` string, date-time, required — When this version was created
      - `created_by_user_id` string, nullable, required — The unique identifier for the user who created this version
      - `type` 'HARNESS', required — Discriminator identifying this as a harness evaluator version.
    - EvaluatorVersionRemote — Evaluator version backed by a remote evaluation config. Only common version metadata (id, commit info, timestamps) is returned — the remote configuration is not yet accessible and will be a future addition.
      - `id` string, required — The unique identifier for this version
      - `evaluator_id` string, required — The parent evaluator ID
      - `commit_hash` string, required — A unique hash identifying this version
      - `commit_message` string, nullable, required — A message describing the changes in this version
      - `created_at` string, date-time, required — When this version was created
      - `created_by_user_id` string, nullable, required — The unique identifier for the user who created this version
      - `type` 'REMOTE', required — Discriminator identifying this as a remote evaluator version.

## Other responses

- `400` — Invalid request
- `401` — Authentication is required
- `403` — Insufficient permissions to access this resource
- `404` — Not found
- `409` — Resource conflict
- `422` — Unprocessable entity
- `429` — Rate limit exceeded

---

[API](https://skmtc.net/arize-ai/apis/arize-rest-api.md) · [All operations](https://skmtc.net/arize-ai/apis/arize-rest-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/arize-ai/arize-rest-api/versions/2ce448f1de13/schema)