Create Evaluation

Create model evaluation job(s) for multiple models and datasets.

Args: request.base_model: Primary model to evaluate (required). request.comparison_models: Optional list of models to compare against. request.dataset_name: Single dataset name (for backward compatibility). request.dataset_names: List of dataset names (creates one evaluation per dataset). request.dataset_version: Optional version (latest if omitted). request.max_tokens: Optional decoder generation cap per example.

This creates evaluation records for each model x dataset combination and triggers async evaluation jobs. Each job runs inference on its dataset and computes metrics.

Returns: EvaluationCreateResponse with list of created evaluations.

Request body

base_modelstring required

Primary model to evaluate (required). Can be a training job ID, explicit GLiNER model ID (e.g. 'fastino/gliner2-base-v1'), decoder base model ID (e.g. 'base:Qwen/Qwen3-8B'), or LLM model ID

comparison_modelsstring[] nullable

Optional list of models to compare against (e.g., ['fastino/gliner2-base-v1', 'gpt-4o']). Backend routing is determined from the model ID.

model_idstring nullable

[DEPRECATED] Use base_model instead. Kept for backward compatibility.

project_idstring nullable

Optional project ID to associate the evaluation with

dataset_namestring nullable

Name of a single dataset to evaluate on (use dataset_names for multiple)

dataset_namesstring[] nullable

List of dataset names to evaluate on (creates one evaluation per dataset)

dataset_versionstring nullable

Dataset version (latest if omitted)

max_examplesinteger nullable

Max examples for LLM evaluation

max_tokensinteger nullable

Optional max generation tokens for decoder evaluations

seedinteger nullable

Random seed for sampling

task_typestring nullable

Task type: 'classification', 'ner', 'json_extraction', or 'decoder'

text_columnstring nullable

Name of the text column

label_columnstring nullable

Name of the label/tags column

Response

Successful Response

successboolean required

countinteger required

v2

Create Evaluation

Request body

Response