---
title: "Create benchmark"
method: POST
path: "/v1/retrievers/benchmarks"
tags: ["Retriever Benchmarks"]
---

# Create benchmark

`POST /v1/retrievers/benchmarks`

Create a new benchmark run to compare retriever pipelines. The benchmark will replay historical sessions and measure alignment with observed user behavior.

## Request body

- CreateBenchmarkRequest — Request to create a new benchmark run.
  - `benchmark_name` string, required — Human-readable name for this benchmark.
  - `baseline_retriever_id` string, required — ID of the baseline retriever pipeline to compare against.
  - `candidate_retriever_ids` string[], required — IDs of candidate retriever pipelines to evaluate.
  - `session_filter` SessionFilterInput — Criteria for selecting historical sessions to replay.
    - `retriever_ids` string[], nullable — Filter to sessions from these retrievers.
    - `taxonomy_node_ids` string[], nullable — Filter to sessions with these taxonomy classifications.
    - `time_range` TimeRangeInput — Time range filter for session queries.
      - `start` string, date-time, nullable — Start of time range (inclusive).
      - `end` string, date-time, nullable — End of time range (inclusive).
    - `min_interactions` integer — Minimum number of user interactions required.
    - `interaction_types` string[], nullable — Filter to sessions with these interaction types (e.g., ['click', 'purchase']).
    - `sample_strategy` string — How to sample sessions: 'random', 'recent', or 'stratified'.
    - `interaction_weights` InteractionWeights — Custom weights for different interaction types when computing metrics. Higher weights indicate more importance. Purchases typically have higher weight than clicks because they're a stronger signal of user intent. Example: {"click": 1.0, "purchase": 5.0, "add_to_cart": 2.0, "bookmark": 1.5}
      - `weights` object — Mapping of interaction_type -> weight (higher = more important).
  - `session_count` integer — Number of sessions to include in the benchmark.

## Response `201`

Successful Response

- BenchmarkResponse — Response containing benchmark details and results.
  - `benchmark_id` string, required — Unique benchmark identifier.
  - `benchmark_name` string, required — Human-readable name.
  - `baseline_retriever_id` string, required — Baseline retriever ID.
  - `candidate_retriever_ids` string[], required — Candidate retriever IDs.
  - `session_filter` SessionFilterOutput — Criteria for selecting historical sessions to replay.
    - `retriever_ids` string[], nullable — Filter to sessions from these retrievers.
    - `taxonomy_node_ids` string[], nullable — Filter to sessions with these taxonomy classifications.
    - `time_range` SharedRetrieversBenchmarksModelsTimeRange — Time range filter for session queries.
      - `start` string, date-time, nullable — Start of time range (inclusive).
      - `end` string, date-time, nullable — End of time range (inclusive).
    - `min_interactions` integer — Minimum number of user interactions required.
    - `interaction_types` string[], nullable — Filter to sessions with these interaction types (e.g., ['click', 'purchase']).
    - `sample_strategy` string — How to sample sessions: 'random', 'recent', or 'stratified'.
    - `interaction_weights` InteractionWeights — Custom weights for different interaction types when computing metrics. Higher weights indicate more importance. Purchases typically have higher weight than clicks because they're a stronger signal of user intent. Example: {"click": 1.0, "purchase": 5.0, "add_to_cart": 2.0, "bookmark": 1.5}
      - `weights` object — Mapping of interaction_type -> weight (higher = more important).
  - `session_count` integer, required — Number of sessions in benchmark.
  - `status` 'pending' | 'building_sessions' | 'replaying' | 'computing_metrics' | 'completed' | 'failed', required — Status of a benchmark run.
  - `results` BenchmarkResult[], nullable — Results per pipeline (available when completed).
    - `retriever_id` string, required — ID of the retriever/pipeline tested.
    - `retriever_name` string, required — Human-readable name of the retriever.
    - `pipeline_hash` string, required — Hash of the pipeline configuration.
    - `metrics` AlignmentMetrics, required — Metrics measuring how well a ranking aligns with observed user behavior. These metrics compare a candidate pipeline's ranking against ground truth derived from actual user interactions (clicks, purchases, etc.).
      - `ndcg_at_k` object, required — Normalized Discounted Cumulative Gain at various K values.
      - `mean_rank_clicked` number, required — Average position of clicked items in the new ranking.
      - `mean_rank_purchased` number, nullable — Average position of purchased items (if any purchases observed).
      - `recall_at_k` object, required — Fraction of interacted items found in top K results.
      - `avg_position_delta` number, required — Average change in position for interacted items (negative = promoted).
      - `items_promoted` integer, required — Number of interacted items moved to higher positions.
      - `items_demoted` integer, required — Number of interacted items moved to lower positions.
      - `sessions_improved` integer, required — Sessions where candidate outperformed baseline.
      - `sessions_degraded` integer, required — Sessions where candidate underperformed baseline.
      - `sessions_neutral` integer, required — Sessions with no significant difference.
    - `taxonomy_deltas` object, nullable — Metrics broken down by taxonomy node (for understanding category-level performance).
    - `latency` LatencyMetrics, required — Performance timing statistics for a pipeline.
      - `p50_ms` number, required — 50th percentile latency in milliseconds.
      - `p90_ms` number, required — 90th percentile latency in milliseconds.
      - `p99_ms` number, required — 99th percentile latency in milliseconds.
      - `mean_ms` number, required — Mean latency in milliseconds.
      - `stage_latencies` object — Per-stage latency breakdown (stage_name -> avg_ms).
    - `failed_sessions` integer, required — Number of sessions that failed during replay.
    - `error_summary` object — Count of errors by type (error_type -> count).
  - `comparison` BenchmarkComparison — unresolved $ref
  - `created_at` string, date-time, required — Creation timestamp.
  - `started_at` string, date-time, nullable — Execution start time.
  - `completed_at` string, date-time, nullable — Completion time.
  - `error_message` string, nullable — Error message if failed.

## Other responses

- `400` — Bad Request
- `401` — Unauthorized
- `403` — Forbidden
- `404` — Not Found
- `422` — Validation Error
- `500` — Internal Server Error

---

[API](https://skmtc.net/mixpeek/apis/mixpeek-api.md) · [All operations](https://skmtc.net/mixpeek/apis/mixpeek-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mixpeek/mixpeek-api/revisions/220a3b263fda/schema)