---
title: "Create an evaluation job"
method: POST
path: "/evaluation"
tags: ["evaluation"]
---

# Create an evaluation job

`POST /evaluation`

## Request body

- EvaluationTypedRequest
  - `type` 'classify' | 'score' | 'compare', required — The type of evaluation to perform
  - `parameters` union, required — Type-specific parameters for the evaluation
    - EvaluationClassifyParameters
      - `judge` EvaluationJudgeModelConfig, required
        - `model` string, required — Name of the judge model
        - `system_template` string, required — System prompt template for the judge
        - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the judge model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs, 20 for proxy/aggregator endpoints.
        - `external_api_token` string — Bearer/API token for the external judge model provider. Required when model_source is 'external'.
        - `external_base_url` string — Base URL of the external inference API for the judge. Must be OpenAI-compatible. Required when model_source is 'external'.
        - `num_workers` integer — Number of concurrent inference workers for the judge. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
        - `max_tokens` integer — Maximum number of tokens the judge model may generate. Defaults to 32768 if omitted. Set higher for reasoning judges (e.g. o-series, Gemini) that spend tokens on internal chain-of-thought before emitting the verdict JSON.
        - `temperature` number — Sampling temperature for the judge model. Defaults to 0.05 if omitted.
      - `labels` string[], required — List of possible classification labels
      - `pass_labels` string[], required — List of labels that are considered passing
      - `model_to_evaluate` union
        - string — Column name in the input dataset containing pre-generated responses
        - EvaluationModelRequest
          - `model` string, required — Name of the model to evaluate
          - `max_tokens` integer, required — Maximum number of tokens to generate.
          - `temperature` number, required — Sampling temperature for generation.
          - `system_template` string, required — System prompt template. Supports Jinja2 variables referencing dataset columns.
          - `input_template` string, required — User message template. Supports Jinja2 variables referencing dataset columns.
          - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). Authentication uses the requesting user's Together API token automatically. - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs (OpenAI, Anthropic, Google), 20 for proxy/aggregator endpoints.
          - `external_api_token` string — Bearer/API token for the external model provider. Required when model_source is 'external'.
          - `external_base_url` string — Base URL of the external inference API. Must be OpenAI-compatible. Required when model_source is 'external'.
          - `num_workers` integer — Number of concurrent inference workers. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
      - `input_data_file_path` string, required — Data file ID
    - EvaluationScoreParameters
      - `judge` EvaluationJudgeModelConfig, required
        - `model` string, required — Name of the judge model
        - `system_template` string, required — System prompt template for the judge
        - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the judge model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs, 20 for proxy/aggregator endpoints.
        - `external_api_token` string — Bearer/API token for the external judge model provider. Required when model_source is 'external'.
        - `external_base_url` string — Base URL of the external inference API for the judge. Must be OpenAI-compatible. Required when model_source is 'external'.
        - `num_workers` integer — Number of concurrent inference workers for the judge. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
        - `max_tokens` integer — Maximum number of tokens the judge model may generate. Defaults to 32768 if omitted. Set higher for reasoning judges (e.g. o-series, Gemini) that spend tokens on internal chain-of-thought before emitting the verdict JSON.
        - `temperature` number — Sampling temperature for the judge model. Defaults to 0.05 if omitted.
      - `min_score` number, required — Minimum possible score
      - `max_score` number, required — Maximum possible score
      - `pass_threshold` number, required — Score threshold for passing
      - `model_to_evaluate` union
        - string — Column name in the input dataset containing pre-generated responses
        - EvaluationModelRequest
          - `model` string, required — Name of the model to evaluate
          - `max_tokens` integer, required — Maximum number of tokens to generate.
          - `temperature` number, required — Sampling temperature for generation.
          - `system_template` string, required — System prompt template. Supports Jinja2 variables referencing dataset columns.
          - `input_template` string, required — User message template. Supports Jinja2 variables referencing dataset columns.
          - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). Authentication uses the requesting user's Together API token automatically. - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs (OpenAI, Anthropic, Google), 20 for proxy/aggregator endpoints.
          - `external_api_token` string — Bearer/API token for the external model provider. Required when model_source is 'external'.
          - `external_base_url` string — Base URL of the external inference API. Must be OpenAI-compatible. Required when model_source is 'external'.
          - `num_workers` integer — Number of concurrent inference workers. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
      - `input_data_file_path` string, required — Data file ID
    - EvaluationCompareParameters
      - `judge` EvaluationJudgeModelConfig, required
        - `model` string, required — Name of the judge model
        - `system_template` string, required — System prompt template for the judge
        - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the judge model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs, 20 for proxy/aggregator endpoints.
        - `external_api_token` string — Bearer/API token for the external judge model provider. Required when model_source is 'external'.
        - `external_base_url` string — Base URL of the external inference API for the judge. Must be OpenAI-compatible. Required when model_source is 'external'.
        - `num_workers` integer — Number of concurrent inference workers for the judge. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
        - `max_tokens` integer — Maximum number of tokens the judge model may generate. Defaults to 32768 if omitted. Set higher for reasoning judges (e.g. o-series, Gemini) that spend tokens on internal chain-of-thought before emitting the verdict JSON.
        - `temperature` number — Sampling temperature for the judge model. Defaults to 0.05 if omitted.
      - `model_a` union — Either an EvaluationModelRequest for generation or a string column name from the dataset (when responses are pre-generated). When both model_a and model_b are EvaluationModelRequest objects, their inference runs execute in parallel to reduce total wall-clock time.
        - EvaluationModelRequest
          - `model` string, required — Name of the model to evaluate
          - `max_tokens` integer, required — Maximum number of tokens to generate.
          - `temperature` number, required — Sampling temperature for generation.
          - `system_template` string, required — System prompt template. Supports Jinja2 variables referencing dataset columns.
          - `input_template` string, required — User message template. Supports Jinja2 variables referencing dataset columns.
          - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). Authentication uses the requesting user's Together API token automatically. - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs (OpenAI, Anthropic, Google), 20 for proxy/aggregator endpoints.
          - `external_api_token` string — Bearer/API token for the external model provider. Required when model_source is 'external'.
          - `external_base_url` string — Base URL of the external inference API. Must be OpenAI-compatible. Required when model_source is 'external'.
          - `num_workers` integer — Number of concurrent inference workers. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
        - string — Column name in the input data containing pre-generated responses
      - `model_b` union — Either an EvaluationModelRequest for generation or a string column name from the dataset (when responses are pre-generated). When both model_a and model_b are EvaluationModelRequest objects, their inference runs execute in parallel to reduce total wall-clock time.
        - EvaluationModelRequest
          - `model` string, required — Name of the model to evaluate
          - `max_tokens` integer, required — Maximum number of tokens to generate.
          - `temperature` number, required — Sampling temperature for generation.
          - `system_template` string, required — System prompt template. Supports Jinja2 variables referencing dataset columns.
          - `input_template` string, required — User message template. Supports Jinja2 variables referencing dataset columns.
          - `model_source` 'serverless' | 'dedicated' | 'external', required — Source of the model inference: - `serverless`: Together's shared serverless inference API. Default concurrency: 25 workers. - `dedicated`: A Together dedicated deployment endpoint. Default concurrency: 5 workers (minimum enforced even if num_workers is set lower). Authentication uses the requesting user's Together API token automatically. - `external`: An external inference API (e.g. OpenAI, Anthropic, Google, OpenRouter). Requires `external_api_token` and `external_base_url`. Default concurrency: 2 workers for first-party APIs (OpenAI, Anthropic, Google), 20 for proxy/aggregator endpoints.
          - `external_api_token` string — Bearer/API token for the external model provider. Required when model_source is 'external'.
          - `external_base_url` string — Base URL of the external inference API. Must be OpenAI-compatible. Required when model_source is 'external'.
          - `num_workers` integer — Number of concurrent inference workers. Overrides the source-specific default (serverless: 25, dedicated: 5, external: 2–20). For dedicated endpoints the value is clamped to a minimum of 5 regardless of what is set here.
        - string — Column name in the input data containing pre-generated responses
      - `input_data_file_path` string, required — Data file ID
      - `disable_position_bias_correction` boolean — When false (default), the judge runs twice per sample: once with model A's response first (original order) and once with model B's response first (flipped order). The two verdicts are reconciled to cancel out position bias. When true, only the original-order pass is run, halving judge cost and latency at the expense of position-bias correction. The result file will not contain flipped-order judge fields when this is true.

## Response `200`

Evaluation job created successfully

- EvaluationResponse
  - `workflow_id` string — The ID of the created evaluation job
  - `status` 'pending' — Initial status of the job

## Other responses

- `400` — Invalid request format
- `500` — Failed to create evaluation job

---

[API](https://skmtc.net/together/apis/together-apis.md) · [All operations](https://skmtc.net/together/apis/together-apis/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/together/together-apis/revisions/468edbdc879c/schema)