Bulk-submit a whole bucket as N auto-chunked, auto-queued batches

One call that streams the bucket's objects (paginated, de-duplicated), chunks them server-side into chunk_size-object batches, and submits each (accept-and-queue). Replaces the client-side chunk loop + cursor pacing — no 409 cursor-overlap, no 5k count-timeout, no manual 429 handling. Poll the returned batch ids / batch_group_id for QUEUED -> PROCESSING.

bucket_identifierstring required

The bucket to ingest.

Request body

collection_idsstring[] nullable

Collections to process. Omit to auto-discover all collections sourced from this bucket (plus downstream dependencies).

chunk_sizeinteger

Objects per batch. Clamped down to your tier's max_batch_size. Default 1000 — a good balance of parallelism and per-batch overhead. Sizing model: each batch runs as its own job whose workers scale at ~1 CPU worker per 500 objects up to a per-job worker ceiling, so 16k-20k objects saturates one job's parallelism — for large corpora (100k+ objects) prefer chunk_size 20000. Larger chunks don't run faster; they only raise the cost of a mid-run failure (progress is still resumable per-object via the processing ledger). Much smaller chunks pay one cluster cold-start each and queue behind your tier's concurrent-batch limit. Full guide: docs.mixpeek.com/operations/batch-ingestion-at-scale.

dedup_strategy'skip' | 'replace' | 'force'

Controls how duplicate objects are handled during batch processing.

Dedup is scoped to (bucket_id, collection_id): an object is considered a duplicate if the target collection already has documents produced from the same source object in any prior batch.

max_objectsinteger nullable

Optional cap on total objects streamed (safety bound for very large buckets). Omit to submit the whole bucket.

Example request

{
  "filters": {
    "AND": [
      {
        "field": "name",
        "operator": "eq",
        "value": "John"
      },
      {
        "field": "age",
        "operator": "gte",
        "value": 30
      }
    ],
    "OR": [
      {
        "field": "status",
        "operator": "eq",
        "value": "active"
      },
      {
        "field": "role",
        "operator": "eq",
        "value": "admin"
      }
    ],
    "NOT": [
      {
        "field": "department",
        "operator": "eq",
        "value": "HR"
      },
      {
        "field": "location",
        "operator": "eq",
        "value": "remote"
      }
    ],
    "case_sensitive": true
  }
}

Response

Successful Response

batch_group_idstring required

Correlates all batches created by this call.

total_objectsinteger required

Distinct objects streamed (post-dedup).

total_batchesinteger required

Batches created.

submittedinteger required

Batches accepted/queued successfully.

failedinteger required

Batches that failed admission/submit.

chunk_sizeinteger required

Effective per-batch object count used.

v5

Bulk-submit a whole bucket as N auto-chunked, auto-queued batches

Path parameters

Request body

Example request

Response