Api Evaluators List

Creating an Evaluator

LLM Evaluators

For LLM evaluators, the frontend should first fetch available evaluation forms from the /eval-forms/ endpoint to get the template configuration, then fill in the form and submit.

Required fields:

• name (str): Display name for the evaluator
• evaluator_slug (str): Unique identifier for the evaluator within the organization
• type (str): llm, human_boolean, human_categorical, human_numerical, human_text
• configurations (dict): Complete evaluation form configuration
• description (str, optional): Description of what this evaluator does
• enabled (bool, optional): Whether the evaluator is active (default: False)

Example request body for LLM evaluator:

{
    "name": "Output Length Checker",
    "type": "llm",
    "description": "Checks if the output meets character count requirements",
    "enabled": true,
    "configurations": {
        "eval_class": "output_char_count",
        "type": "function",
        "note": "",
        "display_name": "Output Character Count",
        "description": "Evaluates the length of the output text",
        "special_fields": [],
        "required_fields": [
            {
                "name": "llm_output",
                "display_name": "LLM Output",
                "type": "textarea",
                "description": "The output text to evaluate",
                "required": true,
                "default_value": null,
                "placeholder": "",
                "choices": [],
                "value": null
            }
        ],
        "inference_filters": [],
        "allow_conditions": true,
        "score_mapping": {
            "primary_score": "output_char_count",
            "secondary_score": null,
            "tertiary_score": null,
            "quaternary_score": null
        },
        "category": "custom"
    }
}

Human Annotation Evaluators

For human annotation evaluators, specify the type and provide choices for categorical evaluators.

Human Boolean Evaluator:

{
    "name": "Quality Check",
    "evaluator_slug": "quality_check",
    "type": "human_boolean",
    "description": "Manual quality assessment"
}

Human Categorical Evaluator:

{
    "name": "Sentiment Rating",
    "evaluator_slug": "sentiment_rating",
    "type": "human_categorical",
    "description": "Manual sentiment classification",
    "categorical_choices": [
        {"name": "Positive", "value": 1},
        {"name": "Neutral", "value": 0},
        {"name": "Negative", "value": -1}
    ]
}

Human Numerical Evaluator:

{
    "name": "Quality Score",
    "evaluator_slug": "quality_score",
    "type": "human_numerical",
    "description": "Rate quality from 1-10"
}

Human Text Evaluator:

{
    "name": "Feedback Comments",
    "evaluator_slug": "feedback_comments",
    "type": "human_text",
    "description": "Detailed feedback comments"
}

Response

Returns the created evaluator with all fields populated, including auto-generated fields like id, created_at, updated_at, and evaluator_slug.

Validation

• For LLM evaluators: The configurations field is validated against the corresponding evaluation form schema from EVAL_FORMS_MAP
• For human categorical evaluators: categorical_choices must be a list of objects with name and value fields
• The eval_class in configurations must exist in the available evaluation forms

Notes

• The organization, created_by, and updated_by fields are automatically set from the authenticated user
• Each evaluator gets a unique evaluator_slug within the organization
• LLM evaluators require a valid eval_class that maps to an available evaluation function