---
title: "Create evaluation dataset"
method: POST
path: "/v1/retrievers/evaluations/datasets"
tags: ["Retriever Evaluations"]
---

# Create evaluation dataset

`POST /v1/retrievers/evaluations/datasets`

Create a ground truth dataset for evaluating retrievers. Include queries with their relevant documents and optional graded relevance scores.

## Request body

- CreateDatasetRequest — Request model for creating a new evaluation dataset.
  - `dataset_name` string, required — Unique name for this dataset
  - `description` string, nullable — Description of what this dataset measures
  - `queries` GroundTruthQuery[], required — List of queries with ground truth relevance labels
    - `query_id` string, required — Unique identifier for this query within the dataset
    - `query_input` object, required — Query input in the same format as retriever execution (e.g., {'text': '...'})
    - `relevant_documents` string[], required — List of feature_ids that are relevant for this query
    - `relevance_scores` object, nullable — Optional graded relevance scores (doc_id -> score, 0-5 where 5 is most relevant). Used for NDCG calculation.
  - `metadata` object, nullable — Additional metadata

## Response `201`

Successful Response

- EvaluationDataset — Complete evaluation dataset with metadata. An evaluation dataset is a collection of queries with ground truth relevance labels, used to measure retriever quality.
  - `dataset_id` string, required — Unique dataset identifier
  - `dataset_name` string, required — Human-readable dataset name
  - `description` string, nullable — Dataset description
  - `queries` GroundTruthQuery[], required — List of queries with ground truth
    - `query_id` string, required — Unique identifier for this query within the dataset
    - `query_input` object, required — Query input in the same format as retriever execution (e.g., {'text': '...'})
    - `relevant_documents` string[], required — List of feature_ids that are relevant for this query
    - `relevance_scores` object, nullable — Optional graded relevance scores (doc_id -> score, 0-5 where 5 is most relevant). Used for NDCG calculation.
  - `created_at` string, date-time, required — When dataset was created
  - `updated_at` string, date-time, required — Last update timestamp
  - `namespace_id` string, required — Namespace this dataset belongs to
  - `internal_id` string, required — Internal organization ID
  - `query_count` integer, required — Number of queries in dataset
  - `metadata` object, nullable — Additional metadata (e.g., labeling instructions, version info)

## Other responses

- `400` — Bad Request
- `401` — Unauthorized
- `403` — Forbidden
- `404` — Not Found
- `422` — Validation Error
- `500` — Internal Server Error

---

[API](https://skmtc.net/mixpeek/apis/mixpeek-api.md) · [All operations](https://skmtc.net/mixpeek/apis/mixpeek-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mixpeek/mixpeek-api/versions/220a3b263fda/schema)