Sign in

v5

latestOpenAPI 3.1.02026-08-025631,1012.8 MB
Collection Taxonomies

Apply Taxonomy to Existing Documents

Apply a taxonomy to all existing documents in a collection retroactively.

This endpoint triggers distributed Ray processing to enrich existing documents with taxonomy data. Unlike automatic materialization (which happens during ingestion), this endpoint allows you to:

  1. Backfill enrichment for documents ingested before the taxonomy was created
  2. Re-apply taxonomy after configuration changes
  3. Process specific subsets using scroll_filters

⚙️ Processing Details:

  • Uses Ray datasets with map_batches for parallel processing
  • Scales horizontally across Ray cluster
  • Non-blocking: Returns immediately with task_id
  • Monitor progress via Tasks API

⚠️ Prerequisites:

  • Taxonomy must exist and be valid
  • Taxonomy must be in collection's taxonomy_applications list
  • Collection must contain documents

📊 Performance:

  • ~1000-5000 docs/second depending on cluster size
  • Parallel processing across multiple Ray workers
  • Batch size and parallelism configurable

🔍 Use Cases:

  • Backfill: Apply new taxonomy to historical data
  • Re-enrichment: Update after taxonomy changes
  • Selective: Process filtered document subsets

See Collections API and Taxonomies API documentation for details.

post/v1/collections/{collection_identifier}/apply-taxonomy

Path parameters

collection_identifierstring required

Collection ID or name to apply taxonomy to

Collection ID or name to apply taxonomy to

Request body

taxonomy_idstring required

ID of the taxonomy to apply. REQUIRED. Must be an existing taxonomy (tax_*). The taxonomy must already be in the collection's taxonomy_applications list.

scroll_filtersobject nullable

Optional filters to limit which documents are enriched. NOT REQUIRED. If not provided, all documents in the collection will be enriched. Use to process specific subsets (e.g., documents missing enrichment).

batch_sizeinteger

Number of documents to process in each parallel batch. NOT REQUIRED. Defaults to 1000. Larger batches = fewer Ray tasks but more memory per task. Smaller batches = more Ray tasks but lower memory per task.

parallelisminteger

Number of parallel Ray workers to use for processing. NOT REQUIRED. Defaults to 4. Higher parallelism = faster processing but more cluster resources. Set based on available Ray cluster capacity.

Example request

{
  "description": "Apply taxonomy to all documents",
  "taxonomy_id": "tax_abc123"
}

Response

Successful Response

task_idstring required

ID of the Ray task executing the materialization

statusstring required

Status of the materialization task

collection_idstring required

Collection ID where taxonomy is being applied

taxonomy_idstring required

Taxonomy ID being applied

estimated_documentsinteger nullable

Estimated number of documents to process (if available)

Example response

{
  "collection_id": "col_products",
  "estimated_documents": 15000,
  "status": "submitted",
  "task_id": "ray_task_abc123",
  "taxonomy_id": "tax_categories"
}