--- title: "List Buckets" method: POST path: "/v1/buckets/list" tags: ["Buckets"] --- # List Buckets `POST /v1/buckets/list` This endpoint lists buckets with pagination, sorting, and filtering options. ## Query parameters - `limit` integer, nullable - `page_size` integer, nullable - `offset` integer, nullable - `page` integer, nullable - `cursor` string, nullable - `next_cursor` string, nullable - `after` string, nullable - `include_total` boolean ## Request body - ListBucketsRequest — Request model for listing buckets. Inherits body-level limit/page_size/offset/page (BACKE-2846). The old hardcoded limit=10/offset=0 defaults moved to the controller so an unset body field can fall back to query-string pagination (body wins when set). - `limit` integer, nullable — Page size. A body value wins over the `limit` query param. - `page_size` integer, nullable — Alias for `limit` (page size). If both are given, `limit` wins. - `offset` integer, nullable — Number of results to skip (legacy, use cursor instead). A body value wins over the `offset` query param. - `page` integer, nullable — 1-indexed page number. Folds to offset = (page-1) * limit. If both `page` and `offset` are given, `offset` wins. - `search` string, nullable — Search term for wildcard search across bucket_id, bucket_name, description, and other text fields - `filters` object, nullable — Filters to apply to the bucket list. Supports filtering by bucket_id or bucket_name. - `sort` object, nullable — Sort options for the bucket list (single-column, legacy) - `sorts` object[], nullable — Sort options for the bucket list as a list of {field, direction} — the shape Studio datatables and the retriever list send. The first entry is the primary sort. Takes precedence over the singular `sort`. - `case_sensitive` boolean — If True, filters and search will be case-sensitive ## Response `200` Successful Response - ListBucketsResponse — Response model for listing buckets. - `results` BucketResponse[], required - `bucket_id` string — Unique identifier for the bucket - `bucket_name` string, required — Human-readable name for the bucket - `description` string, nullable — Description of the bucket - `bucket_schema` BucketSchemaOutput — Schema definition for bucket objects. IMPORTANT: The bucket schema defines what fields your bucket objects will have. This schema is REQUIRED if you want to: 1. Create collections that use input_mappings to process your bucket data 2. Validate object structure before ingestion 3. Enable type-safe data pipelines The schema defines the custom fields that will be used in: - Blob properties (e.g., "content", "thumbnail", "transcript") - Object metadata structure - Blob data structures Example workflow: 1. Create bucket WITH schema defining your data structure 2. Upload objects that conform to that schema 3. Create collections that map schema fields to feature extractors Without a bucket_schema, collections cannot use input_mappings. - `properties` object, required — REQUIRED. Map of field names to their type definitions. Each field must have a 'type' from the supported types: metadata types (string, number, integer, float, boolean, object, array, date, datetime) or file/blob types (text, image, audio, video, pdf, excel). NOTE: Use Mixpeek types, NOT JSON Schema types — e.g. use 'string' not 'keyword', 'text' for text blobs, 'image' for image blobs. Example: {"title": {"type": "string"}, "photo": {"type": "image"}} - `unique_key` UniqueKeyConfig — Configuration for bucket unique key enforcement. Enables automatic uniqueness enforcement on one or more fields from the bucket schema. Supports both single field and compound (multi-field) uniqueness constraints. When configured, the bucket will maintain a lookup table mapping unique key values to document IDs, enabling efficient upsert operations and preventing duplicates. **Impact on Collection Trigger/Re-processing:** When a collection is triggered (POST /collections/{id}/trigger), the unique_key determines whether documents are overwritten or duplicated: - WITH unique_key: Documents get deterministic IDs → re-triggering OVERWRITES existing docs - WITHOUT unique_key: Documents get random IDs → re-triggering CREATES DUPLICATES For idempotent pipelines where re-triggering is safe, configure a unique_key. **Relationship to Extractor position_fields:** The `unique_key` (bucket-level) and `position_fields` (extractor-level) work together to generate deterministic document IDs: - `unique_key`: Identifies unique SOURCE OBJECTS in the bucket (e.g., video_id) - `position_fields`: Identifies unique OUTPUT DOCUMENTS from a single object (e.g., start_time, end_time) Document ID Formula: document_id = hash(source_object_key + extractor_id + collection_id + position_field_values) Example - Processing a 60-second video with 10-second segments: - Bucket unique_key: ["video_id"] → Identifies the source video - Extractor position_fields: ["start_time", "end_time"] → Identifies each segment - Result: 6 unique document IDs (one per segment), all deterministic Without position_fields, all segments would get the SAME document_id and overwrite each other. Without unique_key, reprocessing would create DUPLICATE documents instead of updating. Requirements: - fields: REQUIRED - Array of field names from bucket schema to use as unique constraint - default_policy: OPTIONAL - Bucket-level default insertion policy (can be overridden per request) - All specified fields must exist in the bucket schema - All fields must be scalar types (string, integer, float, uuid) - Field values cannot be null or empty in uploaded objects - Cannot be changed after bucket creation (v1 limitation) Use Cases: - Single field uniqueness: ["video_id"], ["product_sku"], ["user_email"] - Compound uniqueness: ["sensor_id", "timestamp"], ["product_id", "size", "color"] - With default policy: Enables idempotent ingestion without per-request policy - Without default: Requires explicit policy on each upload (safer, more intentional) Insertion Policies: - 'insert': Fail with 409 Conflict if key exists (prevents accidental overwrites) - 'update': Fail with 404 Not Found if key doesn't exist (updates only) - 'upsert': Update if exists, insert if not (idempotent ingestion) Policy Resolution (when uploading objects): 1. Use request-level ?policy= parameter if provided (highest priority) 2. Fall back to bucket-level default_policy if configured 3. Return 400 Bad Request if neither is specified (prevents accidental operations) Examples: Single field with upsert default (idempotent video ingestion): { "fields": ["video_id"], "default_policy": "upsert" } Single field with insert default (prevent duplicate products): { "fields": ["product_sku"], "default_policy": "insert" } Compound fields with upsert (time-series sensor data): { "fields": ["sensor_id", "timestamp"], "default_policy": "upsert" } Compound fields without default (explicit policy required): { "fields": ["user_id", "session_id"] } - `fields` string[], required — Field name(s) from bucket schema to use as unique constraint. REQUIRED - must provide at least one field name. Single field example: ['video_id'] - Enforces uniqueness on video_id alone. Compound example: ['sensor_id', 'timestamp'] - Uniqueness requires BOTH fields to match. All specified fields must: - Exist in the bucket schema - Be scalar types (string, integer, float, uuid - NOT objects or arrays) - Have non-null, non-empty values in all uploaded objects - Be 255 characters or less per string field value Field order doesn't matter (sorted internally for consistency). ['timestamp', 'sensor_id'] is equivalent to ['sensor_id', 'timestamp']. - `default_policy` 'insert' | 'update' | 'upsert', nullable — Default insertion policy for this bucket when not specified per request. OPTIONAL - if omitted, you must provide ?policy= parameter on each upload request. Policies: - 'insert': Create new object only. Fail with 409 Conflict if unique key already exists. Use when: You want to prevent accidental overwrites (safest option). - 'update': Update existing object only. Fail with 404 Not Found if unique key doesn't exist. Use when: You only want to update existing records, never create new ones. - 'upsert': Update if exists, create if not (idempotent operation). Use when: You want idempotent ingestion (re-running is safe). Policy Resolution: 1. Request-level ?policy= parameter takes precedence (if provided) 2. Falls back to this default_policy (if configured) 3. Returns 400 Bad Request if neither is specified Recommendation: Omit default_policy if you want explicit control on each upload. Set default_policy='upsert' for idempotent pipelines. - `metadata` object — Additional metadata for the bucket - `storage_class` 'standard' | 'nearline' | 'coldline' | 'archive' — Provider-agnostic object-storage tier for a bucket (BACKE-2299). The mixpeek API stays provider-agnostic; the object-storage factory maps each value to the underlying provider's equivalent on write (and, where supported, retroactively via lifecycle/rewrite): | mixpeek | GCS | S3 / MinIO | |-----------|-----------|----------------| | standard | STANDARD | STANDARD | | nearline | NEARLINE | STANDARD_IA | | coldline | COLDLINE | GLACIER_IR | | archive | ARCHIVE | GLACIER | Set per-bucket so hot retriever-source buckets stay `standard` while large write-once/read-occasionally media buckets (footage, creatives) opt into a cheaper tier (e.g. ~50% on Nearline for the TS iconik ~13TB footage sync). - `object_count` integer, required — Number of objects in the bucket - `total_size_bytes` integer, required — Total size of all objects in the bucket in bytes - `created_at` string, date-time, nullable — When the bucket was created - `updated_at` string, date-time, nullable — Last modification time of bucket metadata - `last_upload_at` string, date-time, nullable — When the last object was uploaded to this bucket - `stats_updated_at` string, date-time, nullable — When bucket stats were last successfully recalculated - `status` 'PENDING' | 'QUEUED' | 'IN_PROGRESS' | 'PROCESSING' | 'COMPLETED' | 'COMPLETED_WITH_ERRORS' | 'FAILED' | 'CANCELED' | 'INTERRUPTED' | 'UNKNOWN' | 'SKIPPED' | 'DRAFT' | 'ACTIVE' | 'ARCHIVED' | 'SUSPENDED' | 'DEACTIVATED' — Enumeration of task statuses for tracking asynchronous operations. Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution. Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces) Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted ACTIVE: Entity is active and operational (for buckets, collections, etc.) ARCHIVED: Entity has been archived SUSPENDED: Entity has been temporarily suspended Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state. Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately. Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling - `is_locked` boolean — Whether the bucket is locked (read-only) - `batch_stats` BatchStatistics — Statistics about batches in a bucket. - `total` integer — Total number of batches in this bucket - `active` integer — Number of batches that are not completed (DRAFT, PENDING, IN_PROGRESS, PROCESSING) - `completed` integer — Number of completed batches - `failed` integer — Number of failed batches - `storage_stats` StorageStatistics — Statistics about object storage in a bucket. - `total_size_bytes` integer — Total size of all objects/blobs in the bucket in bytes - `avg_size_bytes` integer — Average object size in bytes - `max_size_bytes` integer — Size of the largest object in bytes - `min_size_bytes` integer — Size of the smallest object in bytes - `source_adapter` object, nullable — Source adapter configuration for inbound webhook-driven ingestion - `total_count` integer, required — Total number of buckets matching the query - `pagination` PaginationResponse, required — PaginationResponse. Cursor-based pagination response: - Use next_cursor for navigation - Total count fields only populated when include_total=true - `total` integer, nullable - `page` integer, nullable - `page_size` integer, nullable - `total_pages` integer, nullable - `next_page` string, nullable - `previous_page` string, nullable - `next_cursor` string, nullable - `stats` BucketListStats — Aggregate statistics for a list of buckets. - `total_objects` integer — Total number of objects across all buckets - `total_size_bytes` integer — Total size in bytes across all buckets - `avg_objects_per_bucket` number — Average number of objects per bucket - `avg_size_per_bucket` number — Average size in bytes per bucket ## Other responses - `400` — Bad Request - `401` — Unauthorized - `403` — Forbidden - `404` — Not Found - `422` — Validation Error - `500` — Internal Server Error --- [API](https://skmtc.net/mixpeek/apis/mixpeek-api.md) · [All operations](https://skmtc.net/mixpeek/apis/mixpeek-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/mixpeek/mixpeek-api/versions/220a3b263fda/schema)