--- title: "Create crawl task" method: POST path: "/v1/crawl" tags: ["Crawl"] --- # Create crawl task `POST /v1/crawl` ## Request body - CrawlPayload — Request body model for the /crawl endpoint - `allow_external_links` boolean — Allows the crawler to follow links to external websites. - `allow_subdomains` boolean — Allows the crawler to follow links to subdomains of the main domain. - `callback` union — Webhook configuration for receiving crawl results. - object - `events` string[] - `headers` object - `metadata` object - `url` string, uri, required - string, uri - string, uri - `crawl_entire_domain` boolean — Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. - `exclude_paths` string[] — URL pathname regex patterns that exclude matching URLs from the crawl. - `extract_options` object - `auto_driver_configuration` object — Custom flow for the optimization engine: maps candidate names to the number of attempts to spend on each candidate before advancing (0 skips it). Key order defines the flow order. Providing it opts the request into 'auto' driver selection. - `body` unknown - `browser` union - 'chrome' | 'firefox' — Browser type to emulate - object - `name` 'chrome' | 'firefox', required - `version` string — Specific browser version to emulate - `browser_actions` BrowserAction[] — Array of browser automation actions to execute sequentially - union — A browser automation action. Exactly one action key must be present. - AutoScrollAction — Continuously scroll to load dynamic content - `auto_scroll` union, required - boolean - union — Duration value that accepts various formats. Supports: number (ms), string ("1000"), or string with unit ("2s", "500ms", "2m", "1h") - … - object - … - ClickAction — Click on an element by selector - `click` union, required - union — CSS selector or array of alternative selectors. Use an array when you have multiple possible selectors for the same element. - … - object - … - EvalAction — Execute JavaScript code in page context - `eval` union, required - string - object - … - FetchAction — Make an HTTP request in browser context - `fetch` union, required - string, uri - object - … - FillAction — Fill text into an input field - `fill` union, required — Fill options with mode-specific fields. Use "type" mode for behavioral typing simulation, or "paste" mode for instant paste. - object - … - object - … - GetCookiesAction — Retrieve browser cookies - `get_cookies` union, required - boolean - object - … - GotoAction — Navigate to a URL - `goto` union, required - string, uri - object - … - PressAction — Press a keyboard key - `press` union, required - string - object - … - ScreenshotAction — Capture a page screenshot - `screenshot` union, required - boolean - object - … - ScrollAction — Scroll the page or an element - `scroll` union, required - number - string - object - … - WaitAction — Wait for a specified duration - `wait` union, required - union — Duration value that accepts various formats. Supports: number (ms), string ("1000"), or string with unit ("2s", "500ms", "2m", "1h") - … - object - … - WaitForElementAction — Wait for an element to appear or reach a specific state - `wait_for_element` union, required - union — CSS selector or array of alternative selectors. Use an array when you have multiple possible selectors for the same element. - … - object - … - WaitForNavigationAction — Wait for page navigation to complete - `wait_for_navigation` union, required - 'load' | 'domcontentloaded' | 'networkidle0' | 'networkidle2' - object - … - `city` string — City for geolocation - `consent_header` boolean — Whether to automatically handle cookie consent headers - `cookies` union - object[] — Browser cookies as array of cookie objects - `creation` string, nullable - `domain` string, nullable - `expires` union - string - 'Infinity' - `extensions` string[], nullable - `hostOnly` boolean, nullable - `httpOnly` boolean, nullable - `lastAccessed` string, nullable - `maxAge` union - number - 'Infinity' - '-Infinity' - `name` string - `path` string, nullable - `pathIsDefault` boolean, nullable - `sameSite` 'strict' | 'lax' | 'none' - `secure` boolean - `value` string - string — Browser cookies as string - `country` union — Country code for geolocation and proxy selection - 'AD' | 'AE' | 'AF' | 'AG' | 'AI' | 'AL' | 'AM' | 'AO' | 'AQ' | 'AR' | 'AS' | 'AT' | 'AU' | 'AW' | 'AX' | 'AZ' | 'BA' | 'BB' | 'BD' | 'BE' | 'BF' | 'BG' | 'BH' | 'BI' | 'BJ' | 'BL' | 'BM' | 'BN' | 'BO' | 'BQ' | 'BR' | 'BS' | 'BT' | 'BV' | 'BW' | 'BY' | 'BZ' | 'CA' | 'CC' | 'CD' | 'CF' | 'CG' | 'CH' | 'CI' | 'CK' | 'CL' | 'CM' | 'CN' | 'CO' | 'CR' | 'CU' | 'CV' | 'CW' | 'CX' | 'CY' | 'CZ' | 'DE' | 'DJ' | 'DK' | 'DM' | 'DO' | 'DZ' | 'EC' | 'EE' | 'EG' | 'EH' | 'ER' | 'ES' | 'ET' | 'FI' | 'FJ' | 'FK' | 'FM' | 'FO' | 'FR' | 'GA' | 'GB' | 'GD' | 'GE' | 'GF' | 'GG' | 'GH' | 'GI' | 'GL' | 'GM' | 'GN' | 'GP' | 'GQ' | 'GR' | 'GS' | 'GT' | 'GU' | 'GW' | 'GY' | 'HK' | 'HM' | 'HN' | 'HR' | 'HT' | 'HU' | 'ID' | 'IE' | 'IL' | 'IM' | 'IN' | 'IO' | 'IQ' | 'IR' | 'IS' | 'IT' | 'JE' | 'JM' | 'JO' | 'JP' | 'KE' | 'KG' | 'KH' | 'KI' | 'KM' | 'KN' | 'KP' | 'KR' | 'KW' | 'KY' | 'KZ' | 'LA' | 'LB' | 'LC' | 'LI' | 'LK' | 'LR' | 'LS' | 'LT' | 'LU' | 'LV' | 'LY' | 'MA' | 'MC' | 'MD' | 'ME' | 'MF' | 'MG' | 'MH' | 'MK' | 'ML' | 'MM' | 'MN' | 'MO' | 'MP' | 'MQ' | 'MR' | 'MS' | 'MT' | 'MU' | 'MV' | 'MW' | 'MX' | 'MY' | 'MZ' | 'NA' | 'NC' | 'NE' | 'NF' | 'NG' | 'NI' | 'NL' | 'NO' | 'NP' | 'NR' | 'NU' | 'NZ' | 'OM' | 'PA' | 'PE' | 'PF' | 'PG' | 'PH' | 'PK' | 'PL' | 'PM' | 'PN' | 'PR' | 'PS' | 'PT' | 'PW' | 'PY' | 'QA' | 'RE' | 'RO' | 'RS' | 'RU' | 'RW' | 'SA' | 'SB' | 'SC' | 'SD' | 'SE' | 'SG' | 'SH' | 'SI' | 'SJ' | 'SK' | 'SL' | 'SM' | 'SN' | 'SO' | 'SR' | 'SS' | 'ST' | 'SV' | 'SX' | 'SY' | 'SZ' | 'TC' | 'TD' | 'TF' | 'TG' | 'TH' | 'TJ' | 'TK' | 'TL' | 'TM' | 'TN' | 'TO' | 'TR' | 'TT' | 'TV' | 'TW' | 'TZ' | 'UA' | 'UG' | 'UM' | 'US' | 'UY' | 'UZ' | 'VA' | 'VC' | 'VE' | 'VG' | 'VI' | 'VN' | 'VU' | 'WF' | 'WS' | 'XK' | 'YE' | 'YT' | 'ZA' | 'ZM' | 'ZW' - 'ALL' - `device` 'desktop' | 'mobile' | 'tablet' — Device type for browser emulation - `driver` 'auto' | 'vx6' | 'vx8' | 'vx8-pro' | 'vx10' | 'vx10-pro' | 'vx12' | 'vx12-pro' | 'media-vx6' | 'fast-vx6' — Browser driver to use - `expected_status_codes` integer[] — Expected HTTP status codes for successful requests - `formats` string[] — List of acceptable response formats in order of preference - `headers` object — Custom HTTP headers to include in the request - `http2` boolean — Whether to use HTTP/2 protocol - `is_xhr` boolean — Whether to emulate XMLHttpRequest behavior - `locale` union — Locale for browser language and region settings - 'aa-DJ' | 'aa-ER' | 'aa-ET' | 'af' | 'af-NA' | 'af-ZA' | 'ak' | 'ak-GH' | 'am' | 'am-ET' | 'an-ES' | 'ar' | 'ar-AE' | 'ar-BH' | 'ar-DZ' | 'ar-EG' | 'ar-IN' | 'ar-IQ' | 'ar-JO' | 'ar-KW' | 'ar-LB' | 'ar-LY' | 'ar-MA' | 'ar-OM' | 'ar-QA' | 'ar-SA' | 'ar-SD' | 'ar-SY' | 'ar-TN' | 'ar-YE' | 'as' | 'as-IN' | 'asa' | 'asa-TZ' | 'ast-ES' | 'az' | 'az-AZ' | 'az-Cyrl' | 'az-Cyrl-AZ' | 'az-Latn' | 'az-Latn-AZ' | 'be' | 'be-BY' | 'bem' | 'bem-ZM' | 'ber-DZ' | 'ber-MA' | 'bez' | 'bez-TZ' | 'bg' | 'bg-BG' | 'bho-IN' | 'bm' | 'bm-ML' | 'bn' | 'bn-BD' | 'bn-IN' | 'bo' | 'bo-CN' | 'bo-IN' | 'br-FR' | 'brx-IN' | 'bs' | 'bs-BA' | 'byn-ER' | 'ca' | 'ca-AD' | 'ca-ES' | 'ca-FR' | 'ca-IT' | 'cgg' | 'cgg-UG' | 'chr' | 'chr-US' | 'crh-UA' | 'cs' | 'cs-CZ' | 'csb-PL' | 'cv-RU' | 'cy' | 'cy-GB' | 'da' | 'da-DK' | 'dav' | 'dav-KE' | 'de' | 'de-AT' | 'de-BE' | 'de-CH' | 'de-DE' | 'de-LI' | 'de-LU' | 'dv-MV' | 'dz-BT' | 'ebu' | 'ebu-KE' | 'ee' | 'ee-GH' | 'ee-TG' | 'el' | 'el-CY' | 'el-GR' | 'en' | 'en-AG' | 'en-AS' | 'en-AU' | 'en-BE' | 'en-BW' | 'en-BZ' | 'en-CA' | 'en-DK' | 'en-GB' | 'en-GU' | 'en-HK' | 'en-IE' | 'en-IN' | 'en-JM' | 'en-MH' | 'en-MP' | 'en-MT' | 'en-MU' | 'en-NA' | 'en-NG' | 'en-NZ' | 'en-PH' | 'en-PK' | 'en-SG' | 'en-TT' | 'en-UM' | 'en-US' | 'en-VI' | 'en-ZA' | 'en-ZM' | 'en-ZW' | 'eo' | 'es' | 'es-419' | 'es-AR' | 'es-BO' | 'es-CL' | 'es-CO' | 'es-CR' | 'es-CU' | 'es-DO' | 'es-EC' | 'es-ES' | 'es-GQ' | 'es-GT' | 'es-HN' | 'es-MX' | 'es-NI' | 'es-PA' | 'es-PE' | 'es-PR' | 'es-PY' | 'es-SV' | 'es-US' | 'es-UY' | 'es-VE' | 'et' | 'et-EE' | 'eu' | 'eu-ES' | 'fa' | 'fa-AF' | 'fa-IR' | 'ff' | 'ff-SN' | 'fi' | 'fi-FI' | 'fil' | 'fil-PH' | 'fo' | 'fo-FO' | 'fr' | 'fr-BE' | 'fr-BF' | 'fr-BI' | 'fr-BJ' | 'fr-BL' | 'fr-CA' | 'fr-CD' | 'fr-CF' | 'fr-CG' | 'fr-CH' | 'fr-CI' | 'fr-CM' | 'fr-DJ' | 'fr-FR' | 'fr-GA' | 'fr-GN' | 'fr-GP' | 'fr-GQ' | 'fr-KM' | 'fr-LU' | 'fr-MC' | 'fr-MF' | 'fr-MG' | 'fr-ML' | 'fr-MQ' | 'fr-NE' | 'fr-RE' | 'fr-RW' | 'fr-SN' | 'fr-TD' | 'fr-TG' | 'fur-IT' | 'fy-DE' | 'fy-NL' | 'ga' | 'ga-IE' | 'gd-GB' | 'gez-ER' | 'gez-ET' | 'gl' | 'gl-ES' | 'gsw' | 'gsw-CH' | 'gu' | 'gu-IN' | 'guz' | 'guz-KE' | 'gv' | 'gv-GB' | 'ha' | 'ha-Latn' | 'ha-Latn-GH' | 'ha-Latn-NE' | 'ha-Latn-NG' | 'ha-NG' | 'haw' | 'haw-US' | 'he' | 'he-IL' | 'hi' | 'hi-IN' | 'hne-IN' | 'hr' | 'hr-HR' | 'hsb-DE' | 'ht-HT' | 'hu' | 'hu-HU' | 'hy' | 'hy-AM' | 'id' | 'id-ID' | 'ig' | 'ig-NG' | 'ii' | 'ii-CN' | 'ik-CA' | 'is' | 'is-IS' | 'it' | 'it-CH' | 'it-IT' | 'iu-CA' | 'iw-IL' | 'ja' | 'ja-JP' | 'jmc' | 'jmc-TZ' | 'ka' | 'ka-GE' | 'kab' | 'kab-DZ' | 'kam' | 'kam-KE' | 'kde' | 'kde-TZ' | 'kea' | 'kea-CV' | 'khq' | 'khq-ML' | 'ki' | 'ki-KE' | 'kk' | 'kk-Cyrl' | 'kk-Cyrl-KZ' | 'kk-KZ' | 'kl' | 'kl-GL' | 'kln' | 'kln-KE' | 'km' | 'km-KH' | 'kn' | 'kn-IN' | 'ko' | 'ko-KR' | 'kok' | 'kok-IN' | 'ks-IN' | 'ku-TR' | 'kw' | 'kw-GB' | 'ky-KG' | 'lag' | 'lag-TZ' | 'lb-LU' | 'lg' | 'lg-UG' | 'li-BE' | 'li-NL' | 'lij-IT' | 'lo-LA' | 'lt' | 'lt-LT' | 'luo' | 'luo-KE' | 'luy' | 'luy-KE' | 'lv' | 'lv-LV' | 'mag-IN' | 'mai-IN' | 'mas' | 'mas-KE' | 'mas-TZ' | 'mer' | 'mer-KE' | 'mfe' | 'mfe-MU' | 'mg' | 'mg-MG' | 'mhr-RU' | 'mi-NZ' | 'mk' | 'mk-MK' | 'ml' | 'ml-IN' | 'mn-MN' | 'mr' | 'mr-IN' | 'ms' | 'ms-BN' | 'ms-MY' | 'mt' | 'mt-MT' | 'my' | 'my-MM' | 'nan-TW' | 'naq' | 'naq-NA' | 'nb' | 'nb-NO' | 'nd' | 'nd-ZW' | 'nds-DE' | 'nds-NL' | 'ne' | 'ne-IN' | 'ne-NP' | 'nl' | 'nl-AW' | 'nl-BE' | 'nl-NL' | 'nn' | 'nn-NO' | 'nr-ZA' | 'nso-ZA' | 'nyn' | 'nyn-UG' | 'oc-FR' | 'om' | 'om-ET' | 'om-KE' | 'or' | 'or-IN' | 'os-RU' | 'pa' | 'pa-Arab' | 'pa-Arab-PK' | 'pa-Guru' | 'pa-Guru-IN' | 'pa-IN' | 'pa-PK' | 'pap-AN' | 'pl' | 'pl-PL' | 'ps' | 'ps-AF' | 'pt' | 'pt-BR' | 'pt-GW' | 'pt-MZ' | 'pt-PT' | 'rm' | 'rm-CH' | 'ro' | 'ro-MD' | 'ro-RO' | 'rof' | 'rof-TZ' | 'ru' | 'ru-MD' | 'ru-RU' | 'ru-UA' | 'rw' | 'rw-RW' | 'rwk' | 'rwk-TZ' | 'sa-IN' | 'saq' | 'saq-KE' | 'sc-IT' | 'sd-IN' | 'se-NO' | 'seh' | 'seh-MZ' | 'ses' | 'ses-ML' | 'sg' | 'sg-CF' | 'shi' | 'shi-Latn' | 'shi-Latn-MA' | 'shi-Tfng' | 'shi-Tfng-MA' | 'shs-CA' | 'si' | 'si-LK' | 'sid-ET' | 'sk' | 'sk-SK' | 'sl' | 'sl-SI' | 'sn' | 'sn-ZW' | 'so' | 'so-DJ' | 'so-ET' | 'so-KE' | 'so-SO' | 'sq' | 'sq-AL' | 'sq-MK' | 'sr' | 'sr-Cyrl' | 'sr-Cyrl-BA' | 'sr-Cyrl-ME' | 'sr-Cyrl-RS' | 'sr-Latn' | 'sr-Latn-BA' | 'sr-Latn-ME' | 'sr-Latn-RS' | 'sr-ME' | 'sr-RS' | 'ss-ZA' | 'st-ZA' | 'sv' | 'sv-FI' | 'sv-SE' | 'sw' | 'sw-KE' | 'sw-TZ' | 'ta' | 'ta-IN' | 'ta-LK' | 'te' | 'te-IN' | 'teo' | 'teo-KE' | 'teo-UG' | 'tg-TJ' | 'th' | 'th-TH' | 'ti' | 'ti-ER' | 'ti-ET' | 'tig-ER' | 'tk-TM' | 'tl-PH' | 'tn-ZA' | 'to' | 'to-TO' | 'tr' | 'tr-CY' | 'tr-TR' | 'ts-ZA' | 'tt-RU' | 'tzm' | 'tzm-Latn' | 'tzm-Latn-MA' | 'ug-CN' | 'uk' | 'uk-UA' | 'unm-US' | 'ur' | 'ur-IN' | 'ur-PK' | 'uz' | 'uz-Arab' | 'uz-Arab-AF' | 'uz-Cyrl' | 'uz-Cyrl-UZ' | 'uz-Latn' | 'uz-Latn-UZ' | 'uz-UZ' | 've-ZA' | 'vi' | 'vi-VN' | 'vun' | 'vun-TZ' | 'wa-BE' | 'wae-CH' | 'wal-ET' | 'wo-SN' | 'xh-ZA' | 'xog' | 'xog-UG' | 'yi-US' | 'yo' | 'yo-NG' | 'yue-HK' | 'zh' | 'zh-CN' | 'zh-HK' | 'zh-Hans' | 'zh-Hans-CN' | 'zh-Hans-HK' | 'zh-Hans-MO' | 'zh-Hans-SG' | 'zh-Hant' | 'zh-Hant-HK' | 'zh-Hant-MO' | 'zh-Hant-TW' | 'zh-SG' | 'zh-TW' | 'zu' | 'zu-ZA' - 'auto' - `markdown_backend` 'full_page' | 'main_content' — Selects which markdown conversion strategy to use. "full_page" converts the entire HTML page. "main_content" uses Mozilla Readability to extract the main article content before converting. - `method` 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE' — HTTP method for the request - `network_capture` object[] — Filters for capturing network traffic - `method` 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'CONNECT' | 'OPTIONS' | 'TRACE' | 'PATCH' - `resource_type` union - string — Resource type for network capture filtering - string[] - `status_code` union - number - number[] - `url` object - `type` 'exact' | 'contains' - `value` string, required - `validation` boolean - `wait_for_requests_count` number - `wait_for_requests_count_timeout` number - `os` 'windows' | 'mac os' | 'linux' | 'android' | 'ios' — Operating system to emulate - `parse` boolean — Whether to parse the response content - `parser` union — Custom parser configuration as a key-value map - object - string - `realtime_total_timeout` number — Overall deadline in milliseconds for a realtime request. Clamped to the account total timeout — it can shorten the deadline but never extend it. Has no effect on async requests. - `referrer_type` union — Referrer policy for the request - 'random' | 'no-referer' | 'same-origin' - 'google' | 'bing' | 'facebook' | 'twitter' | 'instagram' - `render` union — Whether to render JavaScript content using a browser - boolean - 'auto' - `request_timeout` number — Request timeout in milliseconds - `session` object - `id` string - `prefetch_userbrowser` boolean - `renew_on_blocked` boolean - `retry` boolean - `timeout` number - `skill` union - string — Skills or capabilities required for the request - string[] — Skills or capabilities required for the request - `state` 'AL' | 'AK' | 'AS' | 'AZ' | 'AR' | 'CA' | 'CO' | 'CT' | 'DE' | 'DC' | 'FL' | 'GA' | 'GU' | 'HI' | 'ID' | 'IL' | 'IN' | 'IA' | 'KS' | 'KY' | 'LA' | 'ME' | 'MD' | 'MA' | 'MI' | 'MN' | 'MS' | 'MO' | 'MT' | 'NE' | 'NV' | 'NH' | 'NJ' | 'NM' | 'NY' | 'NC' | 'ND' | 'MP' | 'OH' | 'OK' | 'OR' | 'PA' | 'PR' | 'RI' | 'SC' | 'SD' | 'TN' | 'TX' | 'UT' | 'VT' | 'VA' | 'VI' | 'WA' | 'WV' | 'WI' | 'WY' — US state for geolocation (only valid when country is US) - `tag` string — User-defined tag for request identification - `url` string — Target URL to scrape - `ignore_query_parameters` boolean — Do not re-scrape the same path with different (or none) query parameters. - `include_paths` string[] — URL pathname regex patterns that include matching URLs in the crawl. - `limit` integer — Maximum number of pages to crawl. - `max_discovery_depth` integer — Maximum depth to crawl based on discovery order. - `name` string — Name of the crawl. - `sitemap` 'skip' | 'include' | 'only' — Sitemap and other methods will be used together to find URLs. - `url` string, required — Url to crawl. ## Response `200` Successful Response - Crawl Task Created - CrawlApiResponse — Crawl API response - `account_name` string, required - `completed` number - `completed_at` union - string - object - `crawl_id` string, uuid, required - `crawl_options` object, required - `allow_external_links` boolean, required - `allow_subdomains` boolean, required - `callback` union - object - `events` string[] - `headers` object - `metadata` object - `url` string, uri, required - string, uri - `crawl_entire_domain` boolean, required - `exclude_paths` string[] - `ignore_query_parameters` boolean, required - `include_paths` string[] - `limit` integer, required - `max_discovery_depth` integer, required - `sitemap` 'skip' | 'include' | 'only', required - `created_at` union, required - string - object - `extract_options` object, nullable - `failed` number - `name` string, nullable - `pending` number - `status` 'queued' | 'running' | 'succeeded' | 'failed' | 'canceled', required - `tasks` object[] - `created_at` string - `status` 'pending' | 'completed' | 'failed', required - `task_id` string, required - `updated_at` string - `total` number - `updated_at` union, required - string - object - `url` string, uri, required --- [API](https://skmtc.net/nimbleway/apis/api-gateway.md) · [All operations](https://skmtc.net/nimbleway/apis/api-gateway/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/nimbleway/api-gateway/revisions/9bfe220b7a29/schema)