Create Batch

Maximum number of objects to include when using filters. If omitted, all matching objects are included (up to 50,000).

Controls how duplicate objects are handled during batch processing.

Dedup is scoped to (bucket_id, collection_id): an object is considered a duplicate if the target collection already has documents produced from the same source object in any prior batch.

Submit the batch for processing immediately after creation — same behavior as the ?auto_submit query parameter (the query parameter wins when both are supplied). Default false: the batch is created in DRAFT status and must be submitted with POST /submit before the draft TTL expires or the reaper cancels it. Accepted in the body because callers kept passing it here and it was silently ignored — the batch parked DRAFT and vanished ~30min later with no signal (FRUSTRATIONS L751).

Deprecated alias of auto_submit (kept for callers taught by older docs/hints). auto_submit wins when both are supplied.

OPTIONAL (auto-generated if not provided). Unique identifier for this batch. Format: 'btch_' prefix followed by 12-character secure token. Generated using generate_secure_token() from shared.utilities.helpers. Used to query batch status and track processing across tiers. Immutable after creation.

REQUIRED. Unique identifier of the bucket containing the objects to process. Must be a valid bucket ID that exists in the system. All object_ids must belong to this bucket. Format: Bucket ID as defined when bucket was created.

Namespace this batch belongs to. Stored at creation time.

Enumeration of task statuses for tracking asynchronous operations.

Task statuses indicate the current state of asynchronous operations like batch processing, object ingestion, clustering, and taxonomy execution.

Status Categories: Operation Statuses: Track progress of async operations Lifecycle Statuses: Track entity state (buckets, collections, namespaces)

Values: PENDING: Task is queued but has not started processing yet IN_PROGRESS: Task is currently being executed PROCESSING: Task is actively processing data (similar to IN_PROGRESS) COMPLETED: Task finished successfully with no errors COMPLETED_WITH_ERRORS: Task finished but some items failed (partial success) FAILED: Task encountered an error and could not complete CANCELED: Task was manually canceled by a user or system UNKNOWN: Task status could not be determined SKIPPED: Task was intentionally skipped DRAFT: Task is in draft state and not yet submitted

ACTIVE: Entity is active and operational (for buckets, collections, etc.)
ARCHIVED: Entity has been archived
SUSPENDED: Entity has been temporarily suspended

Terminal Statuses: COMPLETED, COMPLETED_WITH_ERRORS, FAILED, CANCELED are terminal statuses. Once a task reaches these states, it will not transition to another state.

Partial Success Handling: COMPLETED_WITH_ERRORS indicates that the operation completed but some documents/items failed. The task result includes: - List of successful items - List of failed items with error details - Success rate percentage This allows clients to handle partial success scenarios appropriately.

Polling Guidance: - Poll tasks in PENDING, QUEUED, IN_PROGRESS, or PROCESSING states - Stop polling when task reaches COMPLETED, COMPLETED_WITH_ERRORS, FAILED, or CANCELED - Use exponential backoff (1s → 30s) when polling

Controls how duplicate objects are handled during batch processing.

Dedup is scoped to (bucket_id, collection_id): an object is considered a duplicate if the target collection already has documents produced from the same source object in any prior batch.

key_id of the API key that submitted this batch, resolved server-side at creation. None for async/system-created batches with no request actor context. Audit only.

Display prefix of the submitting API key. Audit only.

Whether the submitting key was server-verified internal AT SUBMIT TIME. None = no actor context (unattributable); never used for billing decisions (those read the live key marker).

OPTIONAL. Legacy error message field for backward compatibility. None if batch succeeded or is still processing. Contains human-readable error description from first failed tier. DEPRECATED: Use tier_tasks[].errors for detailed error information. For multi-tier batches, typically contains the error from the first failed tier. Check tier_tasks array for tier-specific error details and error_summary for aggregation.

OPTIONAL. Human-readable explanation of why the batch failed. None if batch succeeded, is still processing, or is in DRAFT/PENDING state. Populated automatically when a batch transitions to FAILED status. Provides a concise, actionable summary of the root cause. Common reasons include: Ray job failure (spot preemption, OOM, code errors), 0 documents written (processing completed but produced no output), processing stall (no activity detected for extended period), or task exception (submission/validation failures). Use this field for user-facing error displays and alerting.

Batch-level failure classification.

Coarser-grained than ErrorCategory (which classifies individual object errors). FailureCategory is set on the batch itself to tell users why the batch as a whole failed — timeout, infra, orphan, pipeline, or unknown. Drives the "Batch failed: <category>" badge in Studio and lets callers distinguish retryable infra blips from genuine pipeline bugs without parsing human-readable strings.

OPTIONAL. Count of objects that failed during batch processing. Shorthand for len(failed_objects). Stored separately for efficient queries and sorting without loading full failed_objects array.

The type of batch.

OPTIONAL. S3 key where the batch manifest is stored. Contains metadata and row data (Parquet) for Engine processing. For tier 0, points to bucket object manifest. For tier N+, points to collection document manifest. Format: S3 path (e.g., 'namespace_id/internal_id/manifests/tier_0.parquet'). Generated during batch submission.

OPTIONAL. Primary task ID for the batch (typically tier 0 task). Used for backward compatibility with single-tier batch tracking. For multi-tier batches, prefer querying tier_tasks array for granular tracking. Format: Task ID as generated for tier 0.

OPTIONAL. Zero-based index of the currently processing tier. None if batch hasn't started processing (status=DRAFT or PENDING). Updated as batch progresses through tiers. Used to show processing progress: 'Processing tier 2 of 5'. Set to last tier number when batch completes. Example: If processing tier 1 (frames), current_tier=1.

OPTIONAL (defaults to 1). Total number of tiers in the collection DAG. Minimum 1 (tier 0 only = bucket → collection). Set during DAG resolution when batch is submitted. Equals len(dag_tiers) if dag_tiers is populated. Used to calculate progress: current_tier / total_tiers. Example: 5-tier pipeline (bucket → chunks → frames → scenes → summaries) has total_tiers=5.

OPTIONAL (auto-set on creation). ISO 8601 timestamp when batch was created. Set using current_time() from shared.utilities.helpers. Immutable after creation. Used for batch age tracking and cleanup of old batches.

OPTIONAL. Read-time aggregate of documents_written from tier_tasks and extractor_jobs. None means the completion callback has not reported write accounting yet.

OPTIONAL. Read-time aggregate of pages NOT indexed across extractor jobs — multi-page inputs (PDFs) capped by max_document_pages or dropped by per-page failures. Only present when > 0: a COMPLETED batch with pages_dropped > 0 indexed its inputs PARTIALLY. See pages_dropped_reasons; raise the collection's max_document_pages to index more pages.

OPTIONAL. Read-time aggregate of video segments NOT indexed across extractor jobs — videos capped by max_video_segments. Only present when > 0: a COMPLETED batch with segments_dropped > 0 indexed its videos PARTIALLY. See segments_dropped_reasons; raise max_video_segments to cover more of each video.

ALWAYS PRESENT (BACKE-2960). True only after the server CONFIRMED this terminal batch's just-written documents are actually queryable via the filtered document get path: a confirmation, never a prediction, timestamp, or elapsed time. A COMPLETED batch with documents_resolvable=false is still finalizing: its documents may 404 on direct GET even though they are durable (Studio renders status==COMPLETED && documents_resolvable==false as a 'Finalizing, indexing your results' state and keeps polling). Batches created before the feature shipped emit true (historical, long since resolvable); post-ship batches with no stored value emit false (fail toward not-ready). If the bounded post-completion confirm cannot succeed, documents_resolvable_error is set and this flag stays false.

Terminal failure signal for the bounded resolvability confirm (BACKE-2960/BACKE-2987). Set when the post-completion confirm loop could not confirm this batch's documents were queryable within its bound, the case where documents may NEVER become resolvable. When set, documents_resolvable stays false and will not flip: stop polling and surface a 'taking longer than expected' state instead of a spinner. Null while the confirm is pending or after it succeeded.

OPTIONAL. Computed health status for actively processing batches. Only populated when status is PROCESSING or IN_PROGRESS. Values: 'healthy' (recent activity detected), 'stalled' (no activity for 5+ minutes), 'unknown' (no heartbeat data yet). Computed from tier_tasks[].last_activity_at and updated_at. Use this to detect stuck batches before the internal stall detector kills them.

OPTIONAL. Timestamp of the most recent activity across all tier tasks. Aggregated from tier_tasks[].last_activity_at — the latest heartbeat from any tier. Updated approximately every 10 seconds by the BatchJobPoller while processing. A stale value (minutes old) while status is PROCESSING indicates the batch may be stalled. None for batches that have not started processing or have no heartbeat data.

OPTIONAL (defaults to 0). Number of times this batch has been auto-retried due to transient infrastructure failures (spot node preemption, OOM, actor death). Incremented each time the batch is automatically requeued after a retryable failure. User-facing: lets users see that retries happened transparently.

OPTIONAL (defaults to 3). Maximum number of automatic retries for transient failures. When retry_count reaches max_retries, the batch stays in FAILED state. Only transient/infrastructure failures trigger retries — validation and data errors do not.

OPTIONAL. ISO 8601 timestamp of the most recent auto-retry attempt. None if the batch has never been retried. Used to calculate exponential backoff for subsequent retries.

OPTIONAL. Human-readable reason for the most recent auto-retry. None if the batch has never been retried. Describes the transient failure that triggered the retry (e.g., 'Spot node preempted', 'Ray actor died', 'OOM killed').

OPTIONAL. URL to receive an HTTP POST notification when the batch reaches a terminal state (COMPLETED, FAILED, or CANCELED). Set at submit time via SubmitBatchRequest. The webhook is fire-and-forget: delivery failures are logged but never affect batch processing.

OPTIONAL (auto-updated). ISO 8601 timestamp when batch was last modified. Updated using current_time() whenever batch status or tier_tasks change. Used to track batch activity and identify stale batches.

COMPUTED. Human-readable description of the current batch state. Examples: 'Processing 724/50,000 objects (1.4%)', 'Queued — 2 batches ahead', 'Completed in 5m 23s', 'Loading model (stage 1/3)'. Computed on read, not stored in the database.

COMPUTED. Estimated completion timestamp based on current throughput. Derived from progress.eta_seconds + now. None if throughput data is unavailable. Computed on read, not stored in the database.