--- title: "Submit a batch" method: POST path: "/batch/submit" tags: ["Batch"] --- # Submit a batch `POST /batch/submit` Scrape 25K URLs or crawl large websites asynchronously. ## Headers - `Idempotency-Key` string — Any string unique to this submission. Retries with the same key return the original batch. ## Request body - BatchSubmitRequest — Create an asynchronous scraping job. - `input` union, required — Choose a URL list or a site crawl. - object — Scrape up to 25K URLs in one batch. - `mode` 'scrape', required — Scrape the pages in `data.urls`. - `data` union, required — Pages to scrape and their output format. - object — Scrape the listed pages as Markdown. - `format` 'markdown', required — Return page content as Markdown. - `urls` object[], required — Pages to scrape. Maximum 25000. - `url` string, required — Page URL to scrape. - `itemId` string — Your ID for this page, returned with its result. The same URL can use different IDs. - `meta` object — Custom JSON returned unchanged with this page result. - `options` object — Options for Markdown output. - `includeLinks` boolean — Include links in the Markdown. - `includeImages` boolean — Include image references in the Markdown. - `shortenBase64Images` boolean — Shorten inline base64 image data. - `useMainContentOnly` boolean — Return the main content without navigation or footers. - `pdf` object — PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. - … - `includeSelectors` string[], nullable — Keep only the subtrees matching these CSS selectors. Filtered pages are always fetched fresh, ignoring `maxAgeMs`. - `excludeSelectors` string[], nullable — Remove elements matching these CSS selectors. Applied after `includeSelectors`, so an element matching both is removed. - `waitForMs` integer — How long to wait after initial page load, in milliseconds. `0` waits 500 ms. - `settleAnimations` boolean — Wait briefly for CSS and transition animations to settle before extraction, on pages that render in a browser. - `country` 'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw' — Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2). - `maxAgeMs` integer, nullable — Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. - object — Scrape the listed pages as HTML. - `format` 'html', required — Return page content as HTML. - `urls` object[], required — Pages to scrape. Maximum 25000. - `url` string, required — Page URL to scrape. - `itemId` string — Your ID for this page, returned with its result. The same URL can use different IDs. - `meta` object — Custom JSON returned unchanged with this page result. - `options` object — Options for HTML output. - `useMainContentOnly` boolean — Return the main content without navigation or footers. - `pdf` object — PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. - … - `includeSelectors` string[], nullable — Keep only the subtrees matching these CSS selectors. Filtered pages are always fetched fresh, ignoring `maxAgeMs`. - `excludeSelectors` string[], nullable — Remove elements matching these CSS selectors. Applied after `includeSelectors`, so an element matching both is removed. - `waitForMs` integer — How long to wait after initial page load, in milliseconds. `0` waits 500 ms. - `settleAnimations` boolean — Wait briefly for CSS and transition animations to settle before extraction, on pages that render in a browser. - `country` 'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw' — Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2). - `maxAgeMs` integer, nullable — Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. - object — Crawl pages starting from a URL or from a domain's sitemap. - `mode` 'crawl', required — Discover and scrape pages from `data.source`. - `data` union, required — Crawl source and output format. - object — Crawl pages and return Markdown. - `format` 'markdown', required — Return page content as Markdown. - `source` union, required — How to find pages to crawl. - object — Discover pages by following links from one URL. - … - object — Scrape the pages listed in a domain's sitemap. Links on those pages are not followed. - … - `options` object — Options for Markdown output. - `includeLinks` boolean — Include links in the Markdown. - `includeImages` boolean — Include image references in the Markdown. - `shortenBase64Images` boolean — Shorten inline base64 image data. - `useMainContentOnly` boolean — Return the main content without navigation or footers. - `pdf` object — PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. - … - `includeSelectors` string[], nullable — Keep only the subtrees matching these CSS selectors. Filtered pages are always fetched fresh, ignoring `maxAgeMs`. - `excludeSelectors` string[], nullable — Remove elements matching these CSS selectors. Applied after `includeSelectors`, so an element matching both is removed. - `waitForMs` integer — How long to wait after initial page load, in milliseconds. `0` waits 500 ms. - `settleAnimations` boolean — Wait briefly for CSS and transition animations to settle before extraction, on pages that render in a browser. - `country` 'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw' — Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2). - `maxAgeMs` integer, nullable — Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. - object — Crawl pages and return HTML. - `format` 'html', required — Return page content as HTML. - `source` union, required — How to find pages to crawl. - object — Discover pages by following links from one URL. - … - object — Scrape the pages listed in a domain's sitemap. Links on those pages are not followed. - … - `options` object — Options for HTML output. - `useMainContentOnly` boolean — Return the main content without navigation or footers. - `pdf` object — PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. - … - `includeSelectors` string[], nullable — Keep only the subtrees matching these CSS selectors. Filtered pages are always fetched fresh, ignoring `maxAgeMs`. - `excludeSelectors` string[], nullable — Remove elements matching these CSS selectors. Applied after `includeSelectors`, so an element matching both is removed. - `waitForMs` integer — How long to wait after initial page load, in milliseconds. `0` waits 500 ms. - `settleAnimations` boolean — Wait briefly for CSS and transition animations to settle before extraction, on pages that render in a browser. - `country` 'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw' — Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2). - `maxAgeMs` integer, nullable — Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. - `webhookUrl` string — URL notified when the batch finishes. - `tags` string[] — Tags stored on the batch. Filter the batch list by them later. ## Response `202` Batch accepted. Read progress and results from `GET /batch/{batch_id}`. - object - `id` string, required — Batch ID. Poll GET /batch/{batch_id} with it. - `status` 'queued', required — Always `queued`. An accepted batch has not started yet. - `mode` 'scrape' | 'crawl', required — How pages will be selected. - `format` 'markdown' | 'html', required — What each page will be returned as. - `tags` string[], required — Tags stored on the batch. - `crawl` BatchNullableCrawlControls, required — The crawl controls as submitted, so the limits requested can be compared against what the crawl reached. - `source` union, required — Where the crawl started. - object — The crawl discovered pages by following links from one URL. - `type` 'start_url', required - `url` string, required — Page the crawl started from. - object — The crawl scraped the pages listed in the domain's sitemap. - `type` 'sitemap', required - `domain` string, required — Domain whose sitemap supplied the pages. - `max_pages` integer, required — The `maxUrls` submitted with the crawl. A sitemap crawl scrapes only the URLs its sitemap actually lists, up to this many, so `input.reserved` is often lower. - `max_depth` integer, nullable, required — Link depth limit. Always 0 for a sitemap crawl, which never follows links off its URLs; null when a `start_url` crawl set no limit. - `follow_subdomains` boolean, required — Whether links to subdomains were followed. Always false for a sitemap crawl. - `url_pattern` string, nullable, required — RE2 pattern URLs had to match to be crawled. Null when the crawl set none. - `input` BatchIntake, required — What submission took in, and what it charged for. - `reserved` integer, required — Pages credits were reserved for. Everything else — progress, the refund, the completion percentage — is measured against this. - `reserved_is_ceiling` boolean, required — Whether `reserved` is an upper bound the batch may finish under. True only for a crawl that follows links, whose reachable page count is unknowable until it runs. False for a scrape and for a sitemap crawl, where `reserved` is an exact page count. - `submitted` integer, nullable, required — URLs in the list you sent, before validation and de-duplication. Null for a crawl, which is given a source rather than a list. - `duplicates` integer, required — URLs dropped before reserving because another entry resolved to the same page. Non-zero for sitemap crawls too, whose sitemaps routinely list a page more than once. - `invalid` integer, nullable, required — URLs from your list rejected as unusable; the same ones are itemised in `invalid_urls` at submission. Null for a crawl — a crawl that resolves no usable page is rejected outright with a 400 rather than accepted with an empty list. - `credits` object, required — What accepting this batch cost. - `reserved` integer, required — Credits just debited from your balance. Whatever the batch does not spend is refunded when it settles. - `created_at` string, required — When the batch was created. - `invalid_urls` object[], required — Rejected URLs, up to 100. These are not charged. - `url` string, required — Rejected URL. - `reason` string, required — Why it was rejected. - `webhook_secret` string — Signing secret for the completion webhook, returned only here and never again. Store it now; it is not repeated by GET /batch/{batch_id}. - `key_metadata` KeyMetadata — Metadata about the API key used for the request. Included in every response whenever a valid API key is provided, even when the response status is not 200. - `credits_consumed` integer, required — The number of credits consumed by this request. - `credits_remaining` integer, required — The number of credits remaining for your organization after this request. ## Other responses - `400` — Bad request - `401` — Unauthorized - `403` — Concurrent batch limit reached (error_code BATCH_LIMIT_EXCEEDED). - `409` — Idempotency-Key reused with a different body (error_code IDEMPOTENCY_KEY_CONFLICT). - `500` — Batch input could not be staged or queued. --- [API](https://skmtc.net/context/apis/context-dev.md) · [All operations](https://skmtc.net/context/apis/context-dev/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/context/context-dev/versions/182f532c2552/schema)