--- title: "Execute Retriever (Auto-Optimized)" method: POST path: "/v1/retrievers/{retriever_id}/execute" tags: ["Retrievers"] --- # Execute Retriever (Auto-Optimized) `POST /v1/retrievers/{retriever_id}/execute` Execute a retriever and return matching documents. The pipeline is automatically optimized before execution for best performance. **Automatic Optimization:** Your pipeline stages are automatically transformed for optimal performance: - Filters pushed down to reduce expensive operations - Redundant stages merged or eliminated - Grouping operations pushed to database layer (10-100x faster) - Operations reordered for efficiency **Streaming Support:** Set stream=true in the request body to receive real-time stage updates via SSE: - Response uses text/event-stream content type - Each stage emits stage_start and stage_complete events - Final event contains complete results and pagination - Useful for progress tracking and debugging **Response Includes (when stream=false):** - documents: Final matching documents - pagination: Pagination metadata - stage_statistics: Per-stage execution metrics - budget: Credit/time consumption - optimization_applied: Whether optimizations were applied - optimization_summary: Details about transformations (when applied) **Optimization Summary Example:** ```json { "optimization_applied": true, "optimization_summary": { "original_stage_count": 5, "optimized_stage_count": 3, "optimization_time_ms": 8.2, "rules_applied": ["push_down_filters", "group_by_push_down"], "stage_reduction_pct": 40.0 } } ``` Use the /explain endpoint to see the optimized execution plan before running. ## Path parameters - `retriever_id` string, required — Retriever ID or name. Pipeline will be automatically optimized before execution. ## Query parameters - `return_presigned_urls` boolean — Generate presigned URLs for S3-backed blobs and url-shaped fields. Also accepted as a body field — if either source is true, presigning is enabled. - `return_vectors` boolean — Include vector embeddings in result documents. Also accepted as a body field — if either source is true, vectors are returned. - `explain` boolean — Return the inline `explain_plan` (per-stage timings + optimization summary) in the response. Also accepted as a body field — if either source is true, the plan is included. - `include_legacy_results` boolean — DEPRECATED. Restore the legacy `results` alias (a byte-for-byte copy of `documents`). Off by default — `results` previously duplicated the entire document array in every response (~half the payload). Prefer `documents`; use this only as a temporary migration shim. - `skip_cache` boolean — Bypass the execute/stage cache read for this request (a fresh execution; the result is still written to cache). Also accepted as a body field — if either source is true, the cache is skipped. ## Request body - ExecuteRetrieverRequest — Request to execute a retriever with optional streaming support. Inherits all fields from RetrieverExecutionRequest including: - inputs: Runtime input values matching the retriever's input_schema - pagination: Pagination configuration (cursor, offset, etc.) - stream: Enable SSE streaming for real-time stage updates Streaming Execution (stream=True): When streaming is enabled, the response uses Server-Sent Events (SSE) format with Content-Type: text/event-stream. Each stage emits events as it executes: Event Types: - stage_start: Emitted when a stage begins execution - stage_complete: Emitted when a stage finishes with results - stage_error: Emitted if a stage encounters an error - execution_complete: Emitted after all stages finish successfully - execution_error: Emitted if the entire execution fails Each event is a StreamStageEvent containing: - event_type: The type of event - execution_id: Unique execution identifier - stage_name: Human-readable stage name - stage_index: Zero-based stage position - total_stages: Total number of stages - documents: Intermediate results (for stage_complete) - statistics: Stage metrics (duration, counts, etc.) - budget_used: Cumulative resource consumption Example streaming client: ```python response = requests.post( '/v1/retrievers/{id}/execute', json={'inputs': {...}, 'stream': True}, stream=True ) for line in response.iter_lines(): if line.startswith(b'data: '): event = json.loads(line[6:]) print(f"{event['event_type']}: {event.get('stage_name')}") ``` Standard Execution (stream=False, default): Returns a single ExecuteRetrieverResponse with final documents, pagination, and aggregate statistics after all stages complete. - `inputs` object — Runtime inputs for the retriever mapped to the input schema. Keys must match the retriever's input_schema field names. Values depend on field types (text, vector, filters, etc.). REQUIRED unless all retriever inputs have defaults. Common input keys: - 'query': Text search query - 'embedding': Pre-computed vector for search - 'top_k': Number of results to return - 'min_score': Minimum relevance threshold - Any custom fields defined in input_schema **Template Syntax** (Jinja2): Namespaces (uppercase or lowercase): - `INPUT` / `input`: Query inputs (e.g., `{{INPUT.query}}`) - `DOC` / `doc`: Document fields (e.g., `{{DOC.payload.title}}`) - `CONTEXT` / `context`: Execution context - `STAGE` / `stage`: Stage configuration - `SECRET` / `secret`: Vault secrets (e.g., `{{SECRET.api_key}}`) Accessing Data: - Dot notation: `{{DOC.payload.metadata.title}}` - Bracket notation: `{{DOC.payload['special-key']}}` - Array index: `{{DOC.items[0]}}`, `{{DOC.tags[2]}}` - Array first/last: `{{DOC.items | first}}`, `{{DOC.items | last}}` Array Operations: - Iterate: `{% for item in DOC.tags %}{{item}}{% endfor %}` - Extract key: `{{DOC.items | map(attribute='name') | list}}` - Join: `{{DOC.tags | join(', ')}}` - Length: `{{DOC.items | length}}` - Slice: `{{DOC.items[:5]}}` Conditionals: - If: `{% if DOC.status == 'active' %}...{% endif %}` - If-else: `{% if DOC.score > 0.8 %}high{% else %}low{% endif %}` - Ternary: `{{'yes' if DOC.enabled else 'no'}}` Built-in Functions: `max`, `min`, `abs`, `round`, `ceil`, `floor` Custom Filters: `slugify` (URL-safe), `bool` (truthy coercion), `tojson` (JSON encode) S3 URLs: Internal S3 URLs (s3://bucket/key) are automatically presigned when accessed via DOC namespace. - `filters` object, nullable — Optional ad-hoc filters applied at execution time. Merged (AND) with any filters already defined in the retriever's stages. Uses the standard LogicalOperator format: {"AND": [{"field": "brand", "operator": "eq", "value": "Acme"}]}. Supports operators: eq, ne, in, nin, gt, gte, lt, lte, contains, exists, is_null. - `pagination` union — Pagination strategy configuration. When omitted entirely, defaults to cursor-based pagination sized to the pipeline's declared final_top_k (author intent); pipelines that declare no final_top_k default to limit=10. Any explicit pagination (or the deprecated body 'limit') is honored exactly. IMPORTANT: Pagination params do NOT support template variables ({{INPUT.x}} or {{DOCUMENT.x}}). Pagination is a request-level parameter for slicing results, separate from pipeline business logic. Pass cursor/limit values directly from your client code. Cursor values come from the previous response's pagination.cursor field. Supported Methods: - CURSOR (default): Best for infinite scroll, stateless, opaque token - KEYSET: Most efficient, requires stable sort, stateless - OFFSET: Traditional page numbers, can have drift issues - SCROLL: Server-side state, best for bulk exports Use CURSOR for: - Infinite scroll UIs (mobile apps, feeds, timelines) - Real-time updates where consistency matters - When you can't jump to arbitrary pages Use KEYSET for: - Maximum performance with large result sets - Stable sort fields (e.g., score DESC, id ASC) - When you need truly stateless pagination Use OFFSET for: - Traditional page UIs with page numbers - When users need to jump to specific pages - Smaller result sets where drift is acceptable Use SCROLL for: - Bulk exports or processing large datasets - When you need to iterate through all results - Background jobs with progress tracking Example (cursor - first page): {"method": "cursor", "limit": 20, "cursor": null} Example (cursor - next page, using cursor from previous response): {"method": "cursor", "limit": 20, "cursor": "eyJvZmZzZXQiOjIwfQ=="} Example (offset): {"method": "offset", "page_size": 25, "page_number": 2} Example (keyset): {"method": "keyset", "limit": 20, "after": {"score": 0.73, "id": "doc_20"}} - OffsetPaginationParams — Offset-based pagination using page number sizing. Best for: Traditional page UIs with page number navigation How it works: - Uses page numbers (1, 2, 3...) and page size - Calculates offset as: (page_number - 1) * page_size - Simple and familiar for users - Can jump to any page directly Tradeoffs: - Can have "page drift" if data changes between requests - Example: Items added/deleted causes duplicates or gaps - Less efficient for large offsets (database must skip N rows) Use when: - Building traditional page-numbered UIs - Users need to jump to specific pages - Result set is relatively stable - Working with smaller datasets Example: Page 1: {"method": "offset", "page_size": 25, "page_number": 1} Page 2: {"method": "offset", "page_size": 25, "page_number": 2} - `method` 'offset' — Constant identifying offset pagination (REQUIRED). - `page_size` integer — Number of documents per page (REQUIRED). Default: 10. - `page_number` integer — 1-based page index to retrieve (REQUIRED). Default: 1. - CursorPaginationParams — Cursor-based pagination referencing last seen position. Best for: Infinite scroll UIs, mobile apps, real-time feeds How it works: - First request: cursor=null - Response includes next cursor token - Next request: pass cursor from previous response - Stateless: no server-side state - Consistent: no duplicates/gaps even with concurrent writes Use when: - Building infinite scroll interfaces - Users scroll through results sequentially - You need consistency across pages - You don't need to jump to arbitrary pages Example flow: 1. Request: {"method": "cursor", "limit": 20, "cursor": null} 2. Response: {"documents": [...], "pagination": {"cursor": "abc123", "has_next": true}} 3. Request: {"method": "cursor", "limit": 20, "cursor": "abc123"} - `method` 'cursor' — Constant identifying cursor pagination (REQUIRED). - `limit` integer — Maximum number of documents to return per page (REQUIRED). Default: 10. - `cursor` string, nullable — Opaque base64 cursor from previous response (OPTIONAL). null for first page, then use cursor from response.pagination.cursor - ScrollPaginationParams — Scroll-style pagination maintaining server-side context for TTL. Best for: Bulk exports, batch processing, iterating through all results How it works: - Server maintains a snapshot of results - First request: scroll_id=null, returns scroll_id - Subsequent requests: use scroll_id from response - Context expires after scroll_ttl seconds - Consistent view of data (point-in-time snapshot) Tradeoffs: - Requires server-side state (memory/cache) - TTL means sessions can expire - Not suitable for long-lived sessions - Good for background jobs, not user-facing UIs Use when: - Exporting large datasets - Batch processing all results - Background jobs iterating through results - You need consistent point-in-time view Example flow: 1. Request: {"method": "scroll", "limit": 100, "scroll_id": null} 2. Response: {"documents": [...], "scroll_id": "xyz789", "has_next": true} 3. Request: {"method": "scroll", "limit": 100, "scroll_id": "xyz789"} - `method` 'scroll' — Constant identifying scroll pagination (REQUIRED). - `limit` integer — Number of documents to fetch per scroll page (REQUIRED). Default: 100. - `scroll_id` string, nullable — Server-issued scroll session identifier (OPTIONAL). null for first request, then use scroll_id from response - `scroll_ttl` integer — Seconds to keep scroll context alive (REQUIRED). Default: 300 (5 minutes). - KeysetPaginationParams — Stateless keyset pagination relying on last seen sort key. Best for: High-performance pagination, large result sets, stable sorting How it works: - Uses actual field values as pagination markers - Database can use indexes efficiently (WHERE score < 0.73) - No offset calculation or server state - Requires deterministic sort order (e.g., score DESC, id ASC) - Most efficient pagination method Requirements: - Results must be sorted consistently - Sort fields must be in the "after" marker - Example: sorted by (score DESC, id ASC) → after: {score: 0.73, id: "doc_20"} Advantages: - No server-side state (truly stateless) - Consistent even with concurrent writes - Database can use indexes (fast for large datasets) - No offset performance degradation Use when: - You have stable, deterministic sort fields - Working with large result sets (10k+ docs) - Maximum performance is critical - You need infinite scroll with best efficiency Example flow: 1. Request: {"method": "keyset", "limit": 20, "after": null} 2. Response: {"documents": [...], "next_cursor": {"score": 0.85, "id": "doc_20"}} 3. Request: {"method": "keyset", "limit": 20, "after": {"score": 0.85, "id": "doc_20"}} - `method` 'keyset' — Constant identifying keyset pagination (REQUIRED). - `limit` integer — Maximum number of documents to return per page (REQUIRED). Default: 10. - `after` object, nullable — Last seen keyset marker from previous response (OPTIONAL). Must include all sort fields. Example: {'score': 0.73, 'id': 'doc_20'}. null for first page, then use next_cursor from response - `limit` integer, nullable — DEPRECATED alias for the pagination page size — prefer 'pagination' (e.g. {"method": "cursor", "limit": 100}). Previously accepted-and-IGNORED: results silently capped at the default page size (10) regardless of the value, and out-of-range values didn't even 422 (FRUSTRATIONS 2026-07-23). Now: when 'pagination' is absent, 'limit' is honored as the default cursor pagination's page size; when both are provided and disagree, pagination wins and a top-level response warning names the conflict. - `stream` boolean — Enable streaming execution to receive real-time stage updates via Server-Sent Events (SSE). NOT REQUIRED - defaults to False for standard execution. When stream=True: - Response uses text/event-stream content type - Each stage completion emits a StreamStageEvent - Events include: stage_start, stage_complete, stage_error, execution_complete - Clients receive intermediate results and statistics as stages execute - Useful for progress tracking, debugging, and partial result display When stream=False (default): - Response returns after all stages complete - Returns a single RetrieverExecutionResponse with final results - Lower overhead for simple queries Use streaming when: - You want to show real-time progress to users - You need to display intermediate results - Pipeline has many stages or long-running operations - Debugging or monitoring pipeline performance Example streaming client (JavaScript): ```javascript const eventSource = new EventSource('/v1/retrievers/ret_123/execute?stream=true'); eventSource.onmessage = (event) => { const stageEvent = JSON.parse(event.data); if (stageEvent.event_type === 'stage_complete') { console.log(`Stage ${stageEvent.stage_name} completed`); console.log(`Documents: ${stageEvent.documents.length}`); } }; ``` Example streaming client (Python): ```python import requests response = requests.post('/v1/retrievers/ret_123/execute', json={'inputs': {...}, 'stream': True}, stream=True) for line in response.iter_lines(): if line.startswith(b'data: '): event = json.loads(line[6:]) print(f"Stage {event['stage_name']}: {event['event_type']}") ``` - `expand` string[], nullable — OPTIONAL. List of fields containing document IDs to resolve inline. Referenced documents are fetched and attached under an '_expanded' key in each result document. Supports dot-notation for nested fields (e.g., 'items.product_id'). Max 50 unique references per request. Depth is limited to 1 (no recursive expansion). - `skip_cache` boolean — OPTIONAL. Bypass stage result cache for this execution. When True, all stages execute fresh without cache lookup. Useful after corpus updates, retriever config changes, or engine deploys. Results are still written to cache for future requests. - `return_presigned_urls` boolean — Generate presigned URLs for S3-backed blobs and url-shaped fields in result documents. Also accepted as a `return_presigned_urls` query parameter; if either source is true, presigning is enabled. - `return_vectors` boolean — Include vector embeddings in result documents. Also accepted as a `return_vectors` query parameter; if either source is true, vectors are returned. - `write_token` string, nullable — OPTIONAL. Pass the `write_token` returned by a prior direct upsert (options.write_token=true) to get read-your-writes consistency: the read is routed to the primary shard, where your just-written document is immediately searchable, instead of an eventually-consistent replica that can lag several seconds behind. Omit for normal (eventual) reads. - `explain` boolean — OPTIONAL. When true, the response includes `explain_plan` — the query profile for THIS execution: per-stage timings + input/output counts, MVS execution stats, and the optimizer summary (and, once wired, the shard ExecutionTrace: chosen legs, fusion, push-downs, nprobe, served/shadow planner). Analogous to SQL `EXPLAIN ANALYZE` — the query still runs and returns documents; the plan is attached alongside. The same profile is auto-logged for every execution regardless of this flag (so Studio can read it); `explain=true` simply returns it inline in the response. ## Response `200` Execution results with documents, pagination, statistics, and optimization details. When stream=true, returns Server-Sent Events. When stream=false, returns JSON response. - unknown ## Other responses - `400` — Bad Request - `401` — Unauthorized - `403` — Forbidden - `404` — Not Found - `422` — Validation Error - `500` — Internal Server Error --- [API](https://skmtc.net/mixpeek/apis/mixpeek-api.md) · [All operations](https://skmtc.net/mixpeek/apis/mixpeek-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mixpeek/mixpeek-api/revisions/5d4c905106b4/schema)