v5

latestOpenAPI 3.1.02026-08-025631,1012.8 MB

Clusters

Execute Clustering

Execute clustering on a specific cluster.

This endpoint:
1. Validates the cluster exists
2. Queues clustering job via Celery
3. Returns task_id immediately (non-blocking)
4. Celery prepares data and submits to Engine
5. Monitor progress via GET /v1/tasks/{task_id}

Flow:
- API: Receives request
- Celery: Fetches documents, creates parquet, uploads to S3
- Engine: Runs Ray job on parquet data
- Status: Automatically updates cluster when complete

Use GET /v1/clusters/{id}/executions to retrieve results.

Optional body (`ExecuteClusterByIdRequest`) accepts a `filters` override
that applies to this execution only and is not persisted on the cluster.

post/v1/clusters/{cluster_id}/execute

Path parameters

cluster_idstring required

Cluster ID to execute

Request body

mode'full' | 'assign' | 'composite'

Execution mode:

full: Run complete clustering pipeline (default)
assign: Incremental assignment — assign new documents to existing cluster centroids without re-clustering. Requires a previous successful execution with centroids.
composite: Cluster centroids from multiple prior executions to create cross-execution cluster groupings. Requires source_execution_ids pointing to completed runs with centroids.

source_execution_idsstring[] nullable

List of clustering execution run_ids whose centroids will be clustered together (only used when mode='composite'). Each referenced execution must have status='completed' and non-empty centroids. Minimum 2 executions required.

assignment_thresholdnumber nullable

Minimum cosine similarity to assign a document to a cluster (only used when mode='assign'). Documents below this threshold are marked as noise (cluster_id=-1). Range: 0.0-1.0.

sample_sizeinteger nullable

Optional per-execution document cap. When provided, overrides the cluster's stored sample_size for this execution only. The override is not persisted on the cluster. KMeans supports up to 1M; O(N²) algorithms (HDBSCAN, Spectral, Agglomerative) are capped at 50K by the engine.

vis_n_components2 | 3 nullable

Number of dimensions for visualization coordinates (2 or 3). When 3, the z coordinate is populated for depth-based rendering.

defer_visualizationboolean nullable

When True, skip inline UMAP during clustering and compute the scatter coordinates lazily at /visualization time. UMAP is ~90% of the clustering critical path at scale, so this makes large runs complete much faster; the first scatter load pays a one-time bounded cost. The override is not persisted on the cluster.

layout_stability'none' | 'transform' | 'align' nullable

Optional layout-stability override for this execution (LS-5). 'align' registers the new layout onto the previous run's coordinates (default when the cluster stores nothing), 'transform' reuses the previous run's saved projection reducer when compatible, 'none' re-layouts independently. The override is not persisted on the cluster.

Example request

{
  "filters": {
    "AND": [
      {
        "field": "name",
        "operator": "eq",
        "value": "John"
      },
      {
        "field": "age",
        "operator": "gte",
        "value": 30
      }
    ],
    "OR": [
      {
        "field": "status",
        "operator": "eq",
        "value": "active"
      },
      {
        "field": "role",
        "operator": "eq",
        "value": "admin"
      }
    ],
    "NOT": [
      {
        "field": "department",
        "operator": "eq",
        "value": "HR"
      },
      {
        "field": "location",
        "operator": "eq",
        "value": "remote"
      }
    ],
    "case_sensitive": true
  },
  "llm_labeling": {
    "description": "Text-only labeling with multiple fields",
    "enabled": true,
    "include_keywords": true,
    "include_summary": true,
    "labeling_inputs": {
      "input_mappings": [
        {
          "input_key": "title",
          "path": "title",
          "source_type": "payload"
        },
        {
          "input_key": "description",
          "path": "description",
          "source_type": "payload"
        },
        {
          "input_key": "text",
          "path": "text",
          "source_type": "payload"
        }
      ]
    },
    "model_name": "gpt-4o-mini-2024-07-18",
    "provider": "openai"
  }
}

Response

Successful Response

task_idstring required

Unique identifier for the task. REQUIRED. Used to poll task status via GET /v1/tasks/{task_id}. This ID is also stored on parent resources (batches, clusters, etc.) for cross-referencing. Format: UUID v4 or custom string identifier.

Types of asynchronous tasks that can be performed in the system.

Task types identify the specific operation being performed. This helps with tracking, debugging, and filtering tasks by operation type.

Categories: API Tasks: User-initiated operations via API endpoints Engine Tasks: Background processing tasks Inference Tasks: Specialized inference operations

API Task Types: API_NAMESPACES_CREATE: Creating a new namespace API_NAMESPACES_MIGRATIONS_RUN: Running a namespace migration API_BUCKETS_OBJECTS_CREATE: Creating objects in a bucket API_BUCKETS_DELETE: Deleting a bucket and its contents API_BUCKETS_BATCHES_PROCESS: Processing a batch of objects API_BUCKETS_BATCHES_SUBMIT: Submitting a batch for processing API_BUCKETS_UPLOADS_CREATE: Creating an upload session API_BUCKETS_UPLOADS_CONFIRM: Confirming an upload completion API_BUCKETS_UPLOADS_BATCH_CONFIRM: Confirming batch upload completion API_TAXONOMIES_CREATE: Creating a new taxonomy API_TAXONOMIES_EXECUTE: Executing taxonomy classification API_TAXONOMIES_MATERIALIZE: Materializing taxonomy results API_RETRIEVERS_PUBLISH: Publishing retriever assets (OG images, etc.)

Engine Task Types: ENGINE_FEATURE_EXTRACTOR_RUN: Running feature extraction on data ENGINE_INFERENCE_RUN: Running inference operations ENGINE_OBJECT_PROCESSING: Processing object data ENGINE_CLUSTER_BUILD: Building clusters from data

Inference Task Types: THUMBNAIL: Generating thumbnails MATERIALIZE: Materializing processed data

Usage: Task types are automatically assigned when tasks are created. You can filter tasks by type when listing or searching for specific operations.

Enumeration of task statuses for tracking asynchronous operations.

Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution.

Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces)

Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted

ACTIVE: Entity is active and operational (for buckets, collections, etc.)
ARCHIVED: Entity has been archived
SUSPENDED: Entity has been temporarily suspended

Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state.

Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately.

Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling

additional_dataobject nullable

Additional metadata and context for the task. OPTIONAL. Contains job IDs, error details, progress info, and other task-specific metadata.

Common fields (all task types): - 'error': Error message if task failed - 'job_id': Ray job ID for engine tasks - 'from_mongodb': True if retrieved from MongoDB fallback (not Redis)

Batch-specific fields (task_type=api_buckets_batches_process): - 'batch_id': Batch identifier (REQUIRED) - 'bucket_id': Source bucket identifier (REQUIRED) - 'namespace_id': Namespace identifier (REQUIRED) - 'current_tier': Currently processing tier number, 0-indexed (OPTIONAL, None if not started) - 'total_tiers': Total number of tiers in the batch pipeline (REQUIRED) - 'collection_ids': Array of ALL collection IDs across all tiers (REQUIRED) - 'object_count': Number of objects being processed (REQUIRED) - 'sample_object_ids': First 5 object IDs for debugging/display (OPTIONAL)

Performance Note: Full object_ids array is NOT stored in task metadata to avoid bloating task documents (batches with 10k+ objects would add 200KB+ per task). For full object list, query the batch directly via GET /v1/buckets/{bucket_id}/batches/{batch_id}.

Note: For detailed per-tier status, use GET /v1/buckets/{bucket_id}/batches/{batch_id} to access the tier_tasks[] array which contains individual tier statuses, collection_ids, and timestamps for each tier.

errorstring nullable

Flattened error message for convenient error handling. OPTIONAL. Automatically populated from additional_data['error'] when the task has FAILED status. This is a convenience field - the full error details are always available in additional_data['error']. Use this field for displaying errors to users or logging. Will be None if task has not failed or if no error details are available. Serialized as 'error' in API responses for backward compatibility.

queue_positioninteger nullable

1-based position in the Ray processing waitlist. None if the batch was dispatched immediately (no queue). Position 1 means this batch will be processed next.

estimated_wait_minutesnumber nullable

Estimated minutes until this batch starts processing, based on queue position and average batch duration. None if the batch was dispatched immediately.

Example response

{
  "additional_data": {
    "batch_id": "btch_xyz789",
    "bucket_id": "bkt_products",
    "collection_ids": [
      "col_tier0",
      "col_tier1",
      "col_tier2"
    ],
    "current_tier": 1,
    "job_id": "ray_job_123",
    "namespace_id": "ns_abc123",
    "object_count": 10000,
    "sample_object_ids": [
      "obj_001",
      "obj_002",
      "obj_003",
      "obj_004",
      "obj_005"
    ],
    "total_tiers": 3
  },
  "description": "Multi-tier batch processing task in progress (tier 1 of 3) with 10k objects",
  "inputs": [
    "batch_xyz789"
  ],
  "status": "IN_PROGRESS",
  "task_id": "2d322a05-3178-4eca-aac6-b82b0a0313aa",
  "task_type": "api_buckets_batches_process"
}