Sign in

v10

latestOpenAPI 3.1.02026-08-0375143791.1 KB
Batch

Submit a batch

Scrape 25K URLs or crawl large websites asynchronously.

post/batch/submit

Headers

Idempotency-Keystring

Any string unique to this submission. Retries with the same key return the original batch.

Any string unique to this submission. Retries with the same key return the original batch.

Request body

webhookUrlstring

URL notified when the batch finishes.

tagsstring[]

Tags stored on the batch. Filter the batch list by them later.

Example request

{
  "input": {
    "data": {
      "urls": [
        {
          "url": "https://example.com/products/anvil",
          "itemId": "sku-1"
        }
      ],
      "options": {
        "country": "de"
      }
    }
  },
  "tags": [
    "docs",
    "competitor"
  ]
}

Response

Batch accepted. Read progress and results from GET /batch/{batch_id}.

idstring required

Batch ID. Poll GET /batch/{batch_id} with it.

status'queued' required

Always queued. An accepted batch has not started yet.

mode'scrape' | 'crawl' required

How pages will be selected.

format'markdown' | 'html' required

What each page will be returned as.

tagsstring[] required

Tags stored on the batch.

created_atstring required

When the batch was created.

webhook_secretstring

Signing secret for the completion webhook, returned only here and never again. Store it now; it is not repeated by GET /batch/{batch_id}.

Example response

{
  "id": "batch_9f2c8a",
  "tags": [
    "docs"
  ],
  "crawl": {
    "source": {
      "url": "https://example.com/docs"
    },
    "max_pages": 500,
    "url_pattern": "^https://example\\.com/docs/"
  },
  "input": {
    "reserved": 24817,
    "submitted": 25000,
    "duplicates": 183
  },
  "credits": {
    "reserved": 24817
  },
  "invalid_urls": [
    {
      "reason": "Must be a public http:// or https:// URL"
    }
  ]
}