Partially Update Bucket

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Schema definition for bucket objects.

IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to:

1. Create collections that use input_mappings to process your bucket data
2. Validate object structure before ingestion
3. Enable type-safe data pipelines

The schema defines the custom fields that will be used in:

• Blob properties (e.g., "content", "thumbnail", "transcript")
• Object metadata structure
• Blob data structures

Example workflow:

1. Create bucket WITH schema defining your data structure
2. Upload objects that conform to that schema
3. Create collections that map schema fields to feature extractors

Without a bucket_schema, collections cannot use input_mappings.

Configuration for bucket unique key enforcement.

Enables automatic uniqueness enforcement on one or more fields from the bucket schema. Supports both single field and compound (multi-field) uniqueness constraints.

When configured, the bucket will maintain a lookup table mapping unique key values to document IDs, enabling efficient upsert operations and preventing duplicates.

Impact on Collection Trigger/Re-processing: When a collection is triggered (POST /collections/{id}/trigger), the unique_key determines whether documents are overwritten or duplicated:

• WITH unique_key: Documents get deterministic IDs → re-triggering OVERWRITES existing docs
• WITHOUT unique_key: Documents get random IDs → re-triggering CREATES DUPLICATES

For idempotent pipelines where re-triggering is safe, configure a unique_key.

Relationship to Extractor position_fields: The unique_key (bucket-level) and position_fields (extractor-level) work together to generate deterministic document IDs:

• unique_key: Identifies unique SOURCE OBJECTS in the bucket (e.g., video_id)
• position_fields: Identifies unique OUTPUT DOCUMENTS from a single object (e.g., start_time, end_time)

Document ID Formula: document_id = hash(source_object_key + extractor_id + collection_id + position_field_values)

Example - Processing a 60-second video with 10-second segments: - Bucket unique_key: ["video_id"] → Identifies the source video - Extractor position_fields: ["start_time", "end_time"] → Identifies each segment - Result: 6 unique document IDs (one per segment), all deterministic

Without position_fields, all segments would get the SAME document_id and overwrite each other. Without unique_key, reprocessing would create DUPLICATE documents instead of updating.

Requirements: - fields: REQUIRED - Array of field names from bucket schema to use as unique constraint - default_policy: OPTIONAL - Bucket-level default insertion policy (can be overridden per request) - All specified fields must exist in the bucket schema - All fields must be scalar types (string, integer, float, uuid) - Field values cannot be null or empty in uploaded objects - Cannot be changed after bucket creation (v1 limitation)

Use Cases: - Single field uniqueness: ["video_id"], ["product_sku"], ["user_email"] - Compound uniqueness: ["sensor_id", "timestamp"], ["product_id", "size", "color"] - With default policy: Enables idempotent ingestion without per-request policy - Without default: Requires explicit policy on each upload (safer, more intentional)

Insertion Policies: - 'insert': Fail with 409 Conflict if key exists (prevents accidental overwrites) - 'update': Fail with 404 Not Found if key doesn't exist (updates only) - 'upsert': Update if exists, insert if not (idempotent ingestion)

Policy Resolution (when uploading objects): 1. Use request-level ?policy= parameter if provided (highest priority) 2. Fall back to bucket-level default_policy if configured 3. Return 400 Bad Request if neither is specified (prevents accidental operations)

Examples: Single field with upsert default (idempotent video ingestion): { "fields": ["video_id"], "default_policy": "upsert" }

Single field with insert default (prevent duplicate products):
    {
        "fields": ["product_sku"],
        "default_policy": "insert"
    }

Compound fields with upsert (time-series sensor data):
    {
        "fields": ["sensor_id", "timestamp"],
        "default_policy": "upsert"
    }

Compound fields without default (explicit policy required):
    {
        "fields": ["user_id", "session_id"]
    }

Statistics about batches in a bucket.

Statistics about object storage in a bucket.