---
title: "Create a custom LLM-as-a-judge evaluator"
method: POST
path: "/v2/evaluators"
tags: ["evaluators"]
---

# Create a custom LLM-as-a-judge evaluator

`POST /v2/evaluators`

Create a new custom evaluator at the organization level without project or environment bindings. The evaluator uses LLM-as-a-judge to evaluate inputs based on the provided configuration. The response format is automatically generated from the output schema.

## Request body

- RequestCreateCustomEvaluatorInput
  - `description` string
  - `frequency_penalty` number
  - `input_schema` EvaluatorProperty[], required — Schemas
    - `description` string
    - `enum_values` string[] — The allowed values for this property. Only applicable when `type` is `enum`.
    - `label` string
    - `name` string, required
    - `type` 'string' | 'int' | 'float' | 'boolean' | 'enum', required
  - `logprobs` boolean
  - `max_tokens` integer
  - `messages` ModelLLMMessage[], required — LLM Message configuration
    - `content` string
    - `role` string
    - `tool_call_id` string
  - `mode` string
  - `model` string, required — LLM parameters (from BaseLLMConfig)
  - `name` string, required — Basic fields
  - `output_schema` EvaluatorProperty[], required
    - `description` string
    - `enum_values` string[] — The allowed values for this property. Only applicable when `type` is `enum`.
    - `label` string
    - `name` string, required
    - `type` 'string' | 'int' | 'float' | 'boolean' | 'enum', required
  - `presence_penalty` number
  - `provider` string, required — Provider configuration
  - `slug` string
  - `stop` string[]
  - `temperature` number
  - `top_p` number

## Response `200`

OK

- ResponseCreateEvaluatorResponse
  - `bindings` string[]
  - `evaluator_id` string
  - `slug` string
  - `type` 'AGENT_GOAL_ACCURACY' | 'CHAR_COUNT' | 'CHAR_COUNT_RATIO' | 'ANSWER_COMPLETENESS' | 'FAITHFULNESS' | 'JSON_VALIDATOR' | 'PERPLEXITY' | 'PII_DETECTOR' | 'PLACEHOLDER_REGEX' | 'PROFANITY_DETECTOR' | 'SEXISM_DETECTOR' | 'PROMPT_INJECTION' | 'TONE_DETECTION' | 'PROMPT_PERPLEXITY' | 'TOXICITY_DETECTOR' | 'ANSWER_RELEVANCY' | 'ANSWER_CORRECTNESS' | 'REGEX_VALIDATOR' | 'SECRETS_DETECTOR' | 'SQL_VALIDATOR' | 'SEMANTIC_SIMILARITY' | 'TOPIC_ADHERENCE' | 'UNCERTAINTY_DETECTOR' | 'WORD_COUNT' | 'WORD_COUNT_RATIO' | 'INSTRUCTION_ADHERENCE' | 'AGENT_FLOW_QUALITY' | 'AGENT_EFFICIENCY' | 'AGENT_GOAL_COMPLETENESS' | 'CONVERSATION_QUALITY' | 'INTENT_CHANGE' | 'CONTEXT_RELEVANCE' | 'AGENT_TOOL_ERROR_DETECTOR' | 'AGENT_TOOL_TRAJECTORY' | 'HTML_COMPARISON' | 'LLM_AS_A_JUDGE'
  - `version_id` string

## Other responses

- `400` — Bad Request
- `500` — Internal Server Error

---

[API](https://skmtc.net/traceloop/apis/traceloop-api.md) · [All operations](https://skmtc.net/traceloop/apis/traceloop-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/traceloop/traceloop-api/versions/9c12a04a5947/schema)