---
title: "Create Evaluation"
method: POST
path: "/felix/evaluations"
tags: ["felix"]
---

# Create Evaluation

`POST /felix/evaluations`

Create model evaluation job(s) for multiple models and datasets.

Args:
    request.base_model: Primary model to evaluate (required).
    request.comparison_models: Optional list of models to compare against.
    request.dataset_name: Single dataset name (for backward compatibility).
    request.dataset_names: List of dataset names (creates one evaluation per dataset).
    request.dataset_version: Optional version (latest if omitted).
    request.max_tokens: Optional decoder generation cap per example.

This creates evaluation records for each model x dataset combination and triggers
async evaluation jobs. Each job runs inference on its dataset and computes metrics.

Returns:
    EvaluationCreateResponse with list of created evaluations.

## Request body

- EvaluationCreate — Request to create a new model evaluation. Supports both single dataset (dataset_name) and multiple datasets (dataset_names). Supports both single model (base_model) and multiple models (base_model + comparison_models). When multiple datasets/models are provided, one evaluation is created per dataset x model combination.
  - `base_model` string, required — Primary model to evaluate (required). Can be a training job ID, explicit GLiNER model ID (e.g. 'fastino/gliner2-base-v1'), decoder base model ID (e.g. 'base:Qwen/Qwen3-8B'), or LLM model ID
  - `comparison_models` string[], nullable — Optional list of models to compare against (e.g., ['fastino/gliner2-base-v1', 'gpt-4o']). Backend routing is determined from the model ID.
  - `model_id` string, nullable — [DEPRECATED] Use base_model instead. Kept for backward compatibility.
  - `project_id` string, nullable — Optional project ID to associate the evaluation with
  - `dataset_name` string, nullable — Name of a single dataset to evaluate on (use dataset_names for multiple)
  - `dataset_names` string[], nullable — List of dataset names to evaluate on (creates one evaluation per dataset)
  - `dataset_version` string, nullable — Dataset version (latest if omitted)
  - `max_examples` integer, nullable — Max examples for LLM evaluation
  - `max_tokens` integer, nullable — Optional max generation tokens for decoder evaluations
  - `seed` integer, nullable — Random seed for sampling
  - `task_type` string, nullable — Task type: 'classification', 'ner', 'json_extraction', or 'decoder'
  - `text_column` string, nullable — Name of the text column
  - `label_column` string, nullable — Name of the label/tags column

## Response `200`

Successful Response

- EvaluationCreateResponse — Response for creating evaluations (supports single or multiple datasets)
  - `success` boolean, required
  - `evaluations` EvaluationResponse[], required
    - `id` string, required
    - `user_id` string, required
    - `project_id` string, nullable
    - `model_id` string, required
    - `dataset_name` string, required
    - `dataset_version` string, required
    - `model_name` string, nullable
    - `f1_score` number, nullable
    - `precision_score` number, nullable
    - `recall_score` number, nullable
    - `accuracy` number, nullable
    - `validation_loss` number, nullable
    - `subset_accuracy` number, nullable
    - `hamming_loss` number, nullable
    - `bleu_score` number, nullable
    - `rouge_l_score` number, nullable
    - `total_tokens` integer, nullable
    - `total_cost_usd` number, nullable
    - `total_latency_ms` integer, nullable
    - `max_examples` integer, nullable
    - `max_tokens` integer, nullable
    - `seed` integer, nullable
    - `config` object, nullable
    - `status` string, required
    - `job_reference` string, nullable
    - `error_message` string, nullable
    - `error_count` integer, nullable
    - `error_sample` string, nullable
    - `sample_count` integer, nullable
    - `evaluation_time_ms` integer, nullable
    - `created_at` string, date-time, required
    - `completed_at` string, date-time, nullable
    - `failed_examples` object[], nullable
    - `predictions` object[], nullable
  - `count` integer, required

## Other responses

- `422` — Validation Error

---

[API](https://skmtc.net/pioneer/apis/brain-api.md) · [All operations](https://skmtc.net/pioneer/apis/brain-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/pioneer/brain-api/versions/31dfe831e079/schema)