---
title: "Create an Evaluator"
method: POST
path: "/v2/evaluators"
tags: ["Evals"]
---

# Create an Evaluator

`POST /v2/evaluators`

## Request body

- union
  - object
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `output_type` 'boolean' | 'categorical' | 'number' | 'string' — The type of output expected from the evaluator
    - `type` 'llm_eval', required
    - `repetitions` integer, nullable
    - `prompt` string, required
    - `categories` string[], nullable
    - `categorical_labels` object[], nullable
      - `value` string, required
      - `description` string
    - `dataset_id` string, nullable
    - `path` string — Legacy alternative to `project_id`. Storage path whose first segment names the project that owns the evaluator. Mutually exclusive with `project_id`.
    - `project_id` string — Unique identifier of the project that owns the evaluator, as returned by `GET /v2/projects`. Mutually exclusive with `path`.
    - `description` string
    - `key` string, required
    - `mode` 'single', required
    - `model` string, required
  - object
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `output_type` 'boolean' | 'categorical' | 'number' | 'string' — The type of output expected from the evaluator
    - `type` 'llm_eval', required
    - `repetitions` integer, nullable
    - `prompt` string, required
    - `categories` string[], nullable
    - `categorical_labels` object[], nullable
      - `value` string, required
      - `description` string
    - `dataset_id` string, nullable
    - `path` string — Legacy alternative to `project_id`. Storage path whose first segment names the project that owns the evaluator. Mutually exclusive with `project_id`.
    - `project_id` string — Unique identifier of the project that owns the evaluator, as returned by `GET /v2/projects`. Mutually exclusive with `path`.
    - `description` string
    - `key` string, required
    - `mode` 'jury', required
    - `jury` object, required
      - `judges` object[], required
        - `model` string, required
        - `retry` object
          - `count` integer
          - `on_codes` integer[]
        - `fallbacks` object[]
          - `model` string, required
      - `replacement_judges` object[]
        - `model` string, required
        - `retry` object
          - `count` integer
          - `on_codes` integer[]
        - `fallbacks` object[]
          - `model` string, required
      - `min_successful_judges` integer
      - `tie_value` 'Tie'
  - object
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `output_type` 'boolean' | 'number'
    - `code` string, required
    - `type` 'python_eval', required
    - `path` string — Legacy alternative to `project_id`. Storage path whose first segment names the project that owns the evaluator. Mutually exclusive with `project_id`.
    - `project_id` string — Unique identifier of the project that owns the evaluator, as returned by `GET /v2/projects`. Mutually exclusive with `path`.
    - `description` string
    - `key` string, required

## Response `200`

Successfully created an evaluator

- union
  - EvaluatorResponseLlm
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `type` 'llm_eval', required
    - `repetitions` integer, nullable
    - `prompt` string, required
    - `categories` string[], nullable
    - `categorical_labels` object[], nullable
      - `value` string, required
      - `description` string
    - `dataset_id` string, nullable
    - `key` string, required
    - `mode` 'single' | 'jury', required
    - `model` string
    - `jury` object
      - `judges` object[], required
        - `model` string, required
        - `retry` object
          - `count` integer
          - `on_codes` integer[]
        - `fallbacks` object[]
          - `model` string, required
      - `replacement_judges` object[]
        - `model` string, required
        - `retry` object
          - `count` integer
          - `on_codes` integer[]
        - `fallbacks` object[]
          - `model` string, required
      - `min_successful_judges` integer
      - `tie_value` 'Tie'
  - EvaluatorResponseJsonSchema
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `type` 'json_schema', required
    - `schema` string, required
    - `key` string, required
  - EvaluatorResponseHttp
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `type` 'http_eval', required
    - `url` string, required
    - `method` 'GET' | 'POST', required
    - `headers` object, required
    - `payload` object, required
    - `key` string, required
  - EvaluatorResponsePython
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `code` string, required
    - `type` 'python_eval', required
    - `key` string, required
  - EvaluatorResponseFunction
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `type` 'function_eval', required
    - `function_params` union, required
      - object
        - `type` 'contains', required
        - `value` string, required
      - object
        - `type` 'contains_none', required
        - `keywords` string[], required
      - object
        - `type` 'contains_all', required
        - `keywords` string[], required
      - object
        - `type` 'contains_any', required
        - `keywords` string[], required
      - object
        - `type` 'contains_email', required
      - object
        - `type` 'contains_url', required
      - object
        - `type` 'contains_valid_link', required
      - object
        - `type` 'start_with', required
        - `value` string, required
      - object
        - `type` 'ends_with', required
        - `value` string, required
      - object
        - `type` 'exact_match', required
      - object
        - `type` 'length_less_than', required
        - `value` number, required
      - object
        - `type` 'length_between', required
        - `min` number, required
        - `max` number, required
      - object
        - `type` 'length_greater_than', required
        - `value` number, required
      - object
        - `type` 'one_line', required
      - object
        - `type` 'regex', required
        - `pattern` string, required
      - object
        - `type` 'is_valid_json', required
      - object
        - `type` 'moderations_openai', required
      - object
        - `type` 'moderations_google', required
      - object
        - `type` 'bert_score', required
      - object
        - `type` 'bleu_score', required
      - object
        - `type` 'rouge_n', required
      - object
        - `type` 'meteor_score', required
      - object
        - `type` 'cosine_similarity', required
      - object
        - `type` 'levenshtein_distance', required
      - object
        - `type` 'keywords_match', required
        - `keywords` string[], required
      - object
        - `type` 'most_repeated_words', required
      - object
        - `type` 'flesch_reading_ease', required
      - object
        - `type` 'gse_english_level', required
      - object
        - `type` 'words_count', required
      - object
        - `type` 'sentences_count', required
      - object
        - `type` 'lexical_repetition', required
      - object
        - `type` 'grammar_diversity', required
    - `key` string, required
  - EvaluatorResponseRagas
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `type` 'ragas', required
    - `ragas_metric` 'context_precision' | 'context_recall' | 'context_entities_recall' | 'harmfulness' | 'maliciousness' | 'coherence' | 'correctness' | 'conciseness' | 'response_relevancy' | 'faithfulness' | 'summarization' | 'noise_sensitivity', required
    - `key` string, required
    - `model` string, required
  - EvaluatorResponseTypescript
    - `_id` string, required
    - `description` string, required
    - `created` string
    - `updated` string
    - `updated_by_id` string, nullable
    - `project_id` string — Unique identifier of the project owning this evaluator.
    - `guardrail_config` union
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'boolean', required
        - `value` boolean, required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'categorical', required
        - `values` string[], required
      - object
        - `enabled` boolean
        - `alert_on_failure` boolean
        - `type` 'number', required
        - `value` number, required
        - `operator` 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte', required
    - `code` string, required
    - `type` 'typescript_eval', required
    - `key` string, required

## Other responses

- `403` — Domain is not allowed
- `404` — Workspace ID is not found on the request

---

[API](https://skmtc.net/orq-ai/apis/orq-ai-api.md) · [All operations](https://skmtc.net/orq-ai/apis/orq-ai-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/orq-ai/orq-ai-api/revisions/deb8d376faee/schema)