List evaluator templates
Retrieve the built-in LLM-as-a-judge evaluator templates. This is the same catalog the product offers when creating an evaluator, spanning response quality, code quality, trajectory, RAG, security, and session evals.
Each template carries the judge prompt, the labels it returns, the score for each label, its optimization direction, and the granularity it evaluates at.
Creating an evaluator from a template. Pick a template, then call POST /v2/evaluators with its fields mapped onto the request:
| Template field | Where it goes in POST /v2/evaluators |
|---|---|
| column_name | version.template_config.name |
| template | version.template_config.template |
| classification_choices | version.template_config.classification_choices |
| direction | version.template_config.direction |
| data_granularity | version.template_config.data_granularity. Send SPAN, or omit it, when the template's value is null |
| display_name | a label for your own use; reuse it for the evaluator's name or description |
| rails | no destination; classification_choices already carries the same labels |
Then add the fields a template doesn't carry: space_id, name, and type: TEMPLATE on the evaluator; a version.commit_message; and the execution settings template_config.include_explanations, use_function_calling_if_available, and llm_config. Finally, create a task to run the evaluator.
A complete request built from the hallucination template:
{
"space_id": "U3BhY2U6NDkzOkJaSkc=",
"name": "hallucination",
"description": "Built from the hallucination template",
"type": "TEMPLATE",
"version": {
"commit_message": "Initial version from built-in template",
"template_config": {
"name": "hallucination",
"template": "You are evaluating whether an answer is factual given reference text...\n{input}\n{output}",
"classification_choices": { "hallucinated": 1, "factual": 0 },
"direction": "MINIMIZE",
"data_granularity": "SPAN",
"include_explanations": true,
"use_function_calling_if_available": true,
"llm_config": {
"ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
"model_name": "gpt-4o",
"invocation_parameters": { "temperature": 0 },
"provider_parameters": {}
}
}
}
}
Scope: this returns only the built-in catalog, which is identical for every caller and contains no space, account, or user data. It does not include the evaluators that already exist in your space. List those with GET /v2/evaluators.
Pagination: not paginated. The catalog is a small fixed list (28 templates, roughly 32 KB of JSON) and the full set is always returned, so there is no cursor or limit.
<Warning>This endpoint is in alpha, read more here.</Warning>
Response
The full list of built-in evaluator templates.
Example response
{
"evaluator_templates": [
{
"column_name": "hallucination",
"display_name": "Hallucination",
"template": "You are evaluating whether the answer is factual...\n{input}\n{output}",
"rails": [
"hallucinated",
"factual"
],
"classification_choices": {
"hallucinated": 1,
"factual": 0
}
}
]
}