Create Collection

Configuration for collection source (bucket(s) or collection).

Collections can process data from two types of sources:

1. Bucket Source: Process raw objects from one or more buckets (first-stage processing)

   • Use this to create your initial collections from uploaded data
   • Can specify multiple buckets to consolidate data from different sources
   • All buckets must have compatible schemas (validated at creation)
   • Example: Videos from multiple regions → Frame extraction collection

2. Collection Source: Process documents from another collection (decomposition trees)

   • Use this to create multi-stage processing pipelines
   • Example: Frames collection → Scene detection collection

Multi-Bucket Requirements:

• All buckets must have compatible schemas (same fields, types, and required status)
• Schema compatibility is validated when the collection is created
• Documents track which specific bucket they came from via root_bucket_id
• Useful for consolidating data from multiple regions, teams, or environments

The source determines:

• What data the feature extractor receives as input
• The input_schema available for input_mappings and field_passthrough
• The lineage tracking in output documents

Examples: Single bucket: {"type": "bucket", "bucket_ids": ["bkt_products"]} Multi-bucket: {"type": "bucket", "bucket_ids": ["bkt_us", "bkt_eu", "bkt_asia"]} Collection: {"type": "collection", "collection_id": "col_frames"}

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for a feature extractor with field passthrough support.

A feature extractor processes source data (from buckets or collections) and produces features (embeddings, extracted text, detected objects, etc.).

With field passthrough, you can also include selected source fields in the output documents alongside the computed features.

Core Concepts: 1. Feature Extraction: Extractors compute features from input data (e.g., text → embeddings, image → detections, video → scenes) 2. Field Passthrough: Selectively preserve source fields in output (e.g., title, category, campaign_id from source → output documents) 3. Output Schema: Combination of passed-through fields + extractor outputs (e.g., {title, category, text_embedding} all in one document)

How Field Passthrough Works: 1. Define which source fields to include via field_passthrough list 2. During processing, these fields are extracted from source 3. They appear in output documents at root level 4. Combine with extractor outputs for complete documents 5. Use target_path to rename fields for cleaner schemas

Field Selection Modes: - Explicit (field_passthrough + include_all=False): Only listed fields pass through. Clean, controlled output. Example: passthrough=[title, category] → output has ONLY title, category, embedding

- **Inclusive** (include_all=True):
  All source fields pass through, field_passthrough for renaming.
  Example: source has 10 fields → output has all 10 + embedding

- **None** (no field_passthrough):
  Only extractor outputs in documents.
  Example: → output has ONLY embedding (no source fields)

Use Cases: - Preserve Identifiers: Keep campaign_id, product_sku, order_id for tracking - Enable Filtering: Pass category, status, department for query filters - Maintain Context: Include title, description for display - Track Metadata: Preserve author, created_at, source for lineage - Business Logic: Keep priority, region, type for application logic

Common Patterns: 1. Minimal Passthrough (recommended): field_passthrough=[{"source_path": "id"}], include_all=False → Clean output, only ID + extractor features

2. **Metadata Preservation**:
   field_passthrough=[
       {"source_path": "title"},
       {"source_path": "category"},
       {"source_path": "created_at"}
   ]
   → Document has context for display and filtering

3. **Field Renaming**:
   field_passthrough=[
       {"source_path": "doc_title", "target_path": "title"},
       {"source_path": "metadata.author", "target_path": "author"}
   ]
   → Cleaner output schema with flattened fields

4. **Required Fields**:
   field_passthrough=[
       {"source_path": "campaign_id", "required": True},
       {"source_path": "priority", "default": 0}
   ]
   → Ensures critical fields always present

Requirements: - feature_extractor_name: REQUIRED - name of the extractor - version: REQUIRED - extractor version (e.g., "v1") - parameters: NOT REQUIRED - extractor-specific config (model, thresholds, etc.) - input_mappings: NOT REQUIRED - maps extractor inputs to source fields - field_passthrough: NOT REQUIRED - which source fields to preserve (default: none) - include_all_source_fields: NOT REQUIRED - preserve all fields (default: false)

Schedule configuration for automatic collection re-processing.

Attaches a cron or interval schedule to a collection, which creates a COLLECTION_TRIGGER trigger behind the scenes. This is a DX convenience so users don't need to create triggers manually.

Examples: Daily re-crawl at 2am UTC: {"trigger_type": "cron", "schedule_config": {"cron_expression": "0 2 * * *"}}

Every 6 hours:
    {"trigger_type": "interval", "schedule_config": {"interval_seconds": 21600}}

Optional taxonomy applications to automatically enrich documents in this collection. Each taxonomy will classify/enrich documents based on configured retriever matches.

Optional cluster applications to automatically execute when batch processing completes. Each cluster enriches documents with cluster assignments (cluster_id, cluster_label, etc.).

Optional alert applications to automatically execute when documents are ingested. Each alert runs a retriever against new documents and sends notifications if matches are found. Supports both ON_INGEST (triggered per batch) and SCHEDULED (periodic) execution modes.

Optional retriever enrichments to run on documents during post-processing. Each enrichment executes a retriever pipeline and writes selected result fields back to the document. Use for: LLM classification, cross-collection joins, multi-stage enrichment at ingestion time.

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for a feature extractor with field passthrough support.

A feature extractor processes source data (from buckets or collections) and produces features (embeddings, extracted text, detected objects, etc.).

With field passthrough, you can also include selected source fields in the output documents alongside the computed features.

Core Concepts: 1. Feature Extraction: Extractors compute features from input data (e.g., text → embeddings, image → detections, video → scenes) 2. Field Passthrough: Selectively preserve source fields in output (e.g., title, category, campaign_id from source → output documents) 3. Output Schema: Combination of passed-through fields + extractor outputs (e.g., {title, category, text_embedding} all in one document)

How Field Passthrough Works: 1. Define which source fields to include via field_passthrough list 2. During processing, these fields are extracted from source 3. They appear in output documents at root level 4. Combine with extractor outputs for complete documents 5. Use target_path to rename fields for cleaner schemas

Field Selection Modes: - Explicit (field_passthrough + include_all=False): Only listed fields pass through. Clean, controlled output. Example: passthrough=[title, category] → output has ONLY title, category, embedding

- **Inclusive** (include_all=True):
  All source fields pass through, field_passthrough for renaming.
  Example: source has 10 fields → output has all 10 + embedding

- **None** (no field_passthrough):
  Only extractor outputs in documents.
  Example: → output has ONLY embedding (no source fields)

Use Cases: - Preserve Identifiers: Keep campaign_id, product_sku, order_id for tracking - Enable Filtering: Pass category, status, department for query filters - Maintain Context: Include title, description for display - Track Metadata: Preserve author, created_at, source for lineage - Business Logic: Keep priority, region, type for application logic

Common Patterns: 1. Minimal Passthrough (recommended): field_passthrough=[{"source_path": "id"}], include_all=False → Clean output, only ID + extractor features

2. **Metadata Preservation**:
   field_passthrough=[
       {"source_path": "title"},
       {"source_path": "category"},
       {"source_path": "created_at"}
   ]
   → Document has context for display and filtering

3. **Field Renaming**:
   field_passthrough=[
       {"source_path": "doc_title", "target_path": "title"},
       {"source_path": "metadata.author", "target_path": "author"}
   ]
   → Cleaner output schema with flattened fields

4. **Required Fields**:
   field_passthrough=[
       {"source_path": "campaign_id", "required": True},
       {"source_path": "priority", "default": 0}
   ]
   → Ensures critical fields always present

Requirements: - feature_extractor_name: REQUIRED - name of the extractor - version: REQUIRED - extractor version (e.g., "v1") - parameters: NOT REQUIRED - extractor-specific config (model, thresholds, etc.) - input_mappings: NOT REQUIRED - maps extractor inputs to source fields - field_passthrough: NOT REQUIRED - which source fields to preserve (default: none) - include_all_source_fields: NOT REQUIRED - preserve all fields (default: false)

Configuration for collection source (bucket(s) or collection).

Collections can process data from two types of sources:

1. Bucket Source: Process raw objects from one or more buckets (first-stage processing)

   • Use this to create your initial collections from uploaded data
   • Can specify multiple buckets to consolidate data from different sources
   • All buckets must have compatible schemas (validated at creation)
   • Example: Videos from multiple regions → Frame extraction collection

2. Collection Source: Process documents from another collection (decomposition trees)

   • Use this to create multi-stage processing pipelines
   • Example: Frames collection → Scene detection collection

Multi-Bucket Requirements:

• All buckets must have compatible schemas (same fields, types, and required status)
• Schema compatibility is validated when the collection is created
• Documents track which specific bucket they came from via root_bucket_id
• Useful for consolidating data from multiple regions, teams, or environments

The source determines:

• What data the feature extractor receives as input
• The input_schema available for input_mappings and field_passthrough
• The lineage tracking in output documents

Examples: Single bucket: {"type": "bucket", "bucket_ids": ["bkt_products"]} Multi-bucket: {"type": "bucket", "bucket_ids": ["bkt_us", "bkt_eu", "bkt_asia"]} Collection: {"type": "collection", "collection_id": "col_frames"}

NOT REQUIRED (auto-computed). Lineage chain showing complete processing history. Each entry contains: source_config, feature_extractor, output_schema for one tier. Length indicates processing depth (1 = tier 1, 2 = tier 2, etc.). Use for: Understanding multi-tier pipelines, visualizing decomposition trees.

NOT REQUIRED (auto-computed from extractor). Vector indexes for semantic search. Populated from feature_extractor.required_vector_indexes. Defines: Which embeddings are indexed, dimensions, distance metrics. Use for: Understanding search capabilities, debugging vector queries.

NOT REQUIRED (auto-computed from extractor + namespace). Payload indexes for filtering. Enables efficient filtering on metadata fields, timestamps, IDs. Populated from: extractor requirements + namespace defaults. Use for: Understanding which fields support fast filtering.

Schedule configuration for automatic collection re-processing.

Attaches a cron or interval schedule to a collection, which creates a COLLECTION_TRIGGER trigger behind the scenes. This is a DX convenience so users don't need to create triggers manually.

Examples: Daily re-crawl at 2am UTC: {"trigger_type": "cron", "schedule_config": {"cron_expression": "0 2 * * *"}}

Every 6 hours:
    {"trigger_type": "interval", "schedule_config": {"interval_seconds": 21600}}

Automatic storage tiering rules (not enforced in V1).

NOT REQUIRED. List of taxonomies to apply to documents in this collection. Each entry specifies: taxonomy_id, optional target_collection_id, optional filters. Enrichments are materialized (persisted to documents) during ingestion. Empty/null if no taxonomies attached. Use for: Categorization, hierarchical classification.

NOT REQUIRED. List of clusters to automatically execute when batch processing completes. Each entry specifies: cluster_id, auto_execute_on_batch, min_document_threshold, cooldown_seconds. Clusters enrich source documents with cluster assignments (cluster_id, cluster_label, etc.). Empty/null if no clusters attached. Use for: Segmentation, grouping, pattern discovery.

NOT REQUIRED. List of alerts to automatically execute when documents are ingested. Each entry specifies: alert_id, execution_mode, input_mappings, execution_phase, priority. Alerts run retrievers on ingested documents and send notifications when matches are found. Empty/null if no alerts attached. Use for: Content monitoring, safety detection, compliance alerts.

NOT REQUIRED. List of retriever enrichments to run on documents during post-processing. Each entry specifies: retriever_id, input_mappings, write_back_fields, execution_phase, priority. Retriever enrichments execute a retriever pipeline and write selected result fields back to documents. Empty/null if no enrichments attached. Use for: LLM classification, cross-collection joins, enrichment.