--- title: "Crawl Website & Scrape Markdown" method: POST path: "/web/crawl" tags: ["Web Scraping"] --- # Crawl Website & Scrape Markdown `POST /web/crawl` Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages. ## Request body - object - `url` string, uri, required — The starting URL for the crawl (must include http:// or https:// protocol) - `maxPages` integer — Maximum number of pages to crawl. Hard cap: 500. - `maxDepth` integer — Maximum link depth from the starting URL (0 = only the starting page) - `urlRegex` string — Regex pattern. Only URLs matching this pattern will be followed and scraped. - `includeLinks` boolean — Preserve hyperlinks in the Markdown output - `includeImages` boolean — Include image references in the Markdown output - `shortenBase64Images` boolean — Truncate base64-encoded image data in the Markdown output - `useMainContentOnly` boolean — Extract only the main content, stripping headers, footers, sidebars, and navigation - `followSubdomains` boolean — When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent. - `pdf` object — PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. - `shouldParse` boolean — When true, PDF pages are fetched and parsed. When false, PDF pages are skipped entirely (not included in results and not counted as failures). - `ocr` boolean — When true, OCR the selected PDF pages that have no usable text layer (scans), replacing each recovered page's text with the OCR result while pages with a real text layer keep it. Billed at 1 credit per page OCR actually recovered, on top of the base request cost. - `start` integer — First 1-based PDF page to parse. When omitted, parsing starts at the first page. - `end` integer — Last 1-based PDF page to parse. When omitted, parsing ends at the last page. Must be greater than or equal to start when both are provided. - `includeFrames` boolean — When true, the contents of iframes are rendered to Markdown for each crawled page. - `includeSelectors` string[] — CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each crawled page is converted to Markdown. When omitted, the entire document is kept. Examples: "article.main", "#content", "[role=main]". - `excludeSelectors` string[] — CSS selectors to remove before each crawled page is converted to Markdown. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: "nav", "footer", ".ad-banner", "[aria-hidden=true]". - `maxAgeMs` integer — Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. - `waitForMs` integer — Optional browser wait time in milliseconds after initial page load for each crawled page. Min: 0. Max: 30000 (30 seconds). - `settleAnimations` boolean — When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages. - `stopAfterMs` integer — Soft time budget for the crawl in milliseconds. After each scrape, the crawler checks the elapsed time and, if exceeded, returns the pages collected so far instead of continuing. Min: 10000 (10s). Max: 110000 (110s). Default: 80000 (80s). - `country` 'ad' | 'ae' | 'af' | 'ag' | 'ai' | 'al' | 'am' | 'ao' | 'ar' | 'at' | 'au' | 'aw' | 'az' | 'ba' | 'bb' | 'bd' | 'be' | 'bf' | 'bg' | 'bh' | 'bi' | 'bj' | 'bm' | 'bn' | 'bo' | 'bq' | 'br' | 'bs' | 'bw' | 'by' | 'bz' | 'ca' | 'cd' | 'cf' | 'cg' | 'ch' | 'ci' | 'cl' | 'cm' | 'cn' | 'co' | 'cr' | 'cv' | 'cw' | 'cy' | 'cz' | 'de' | 'dj' | 'dk' | 'dm' | 'do' | 'dz' | 'ec' | 'ee' | 'eg' | 'es' | 'et' | 'fi' | 'fj' | 'fr' | 'ga' | 'gb' | 'gd' | 'ge' | 'gf' | 'gg' | 'gh' | 'gm' | 'gn' | 'gp' | 'gq' | 'gr' | 'gt' | 'gu' | 'gw' | 'gy' | 'hk' | 'hn' | 'hr' | 'ht' | 'hu' | 'id' | 'ie' | 'il' | 'im' | 'in' | 'iq' | 'ir' | 'is' | 'it' | 'je' | 'jm' | 'jo' | 'jp' | 'ke' | 'kg' | 'kh' | 'kn' | 'kr' | 'kw' | 'ky' | 'kz' | 'la' | 'lb' | 'lc' | 'lk' | 'lr' | 'ls' | 'lt' | 'lu' | 'lv' | 'ly' | 'ma' | 'mc' | 'md' | 'me' | 'mf' | 'mg' | 'mk' | 'ml' | 'mm' | 'mn' | 'mo' | 'mq' | 'mr' | 'mt' | 'mu' | 'mv' | 'mw' | 'mx' | 'my' | 'mz' | 'na' | 'nc' | 'ne' | 'ng' | 'ni' | 'nl' | 'no' | 'np' | 'nz' | 'om' | 'pa' | 'pe' | 'pf' | 'pg' | 'ph' | 'pk' | 'pl' | 'pr' | 'ps' | 'pt' | 'py' | 'qa' | 're' | 'ro' | 'rs' | 'ru' | 'rw' | 'sa' | 'sc' | 'sd' | 'se' | 'sg' | 'si' | 'sk' | 'sl' | 'sm' | 'sn' | 'so' | 'sr' | 'ss' | 'st' | 'sv' | 'sx' | 'sy' | 'sz' | 'tc' | 'td' | 'tg' | 'th' | 'tj' | 'tl' | 'tm' | 'tn' | 'tr' | 'tt' | 'tw' | 'tz' | 'ua' | 'ug' | 'us' | 'uy' | 'uz' | 'vc' | 've' | 'vg' | 'vi' | 'vn' | 'ye' | 'yt' | 'za' | 'zm' | 'zw' — Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2). - `timeoutMS` integer — Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes). - `zdr` 'enabled' | 'disabled' — Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true. - `tags` string[] — Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters. ## Response `200` Successful response - object - `results` object[], required - `markdown` string, required — Extracted page content as Markdown (empty string on failure) - `metadata` object, required — Metadata extracted from the scraped page HTML. - `sourceUrl` string, required — Original URL requested by the caller. - `finalUrl` string, required — Final URL scraped after redirects or scraper fallback, when known. Falls back to sourceUrl when unavailable. - `title` string, required — Best page title extracted from the page (empty string if unavailable). - `description` string — Best description extracted from standard, Open Graph, or Twitter metadata. - `language` string — Language extracted from html lang or language meta tags. - `keywords` string[] — Keywords extracted from the page's keywords meta tag. - `canonicalUrl` string — Resolved canonical URL, when present. - `author` string — Author metadata, when present. - `siteName` string — Site or application name from page metadata. - `image` string — Primary resolved preview image from Open Graph, Twitter, or image metadata. - `favicon` string — Resolved favicon URL, when present. - `publishedTime` string — Published timestamp/date from page metadata, when present. - `modifiedTime` string — Modified timestamp/date from page metadata, when present. - `robots` string — Robots meta directive, when present. - `openGraph` object — Open Graph metadata with the og: prefix removed and keys camel-cased. - `twitter` object — Twitter card metadata with the twitter: prefix removed and keys camel-cased. - `alternates` PageMetadataAlternate[] — Resolved alternate links from link rel=alternate tags. - `href` string, required — Resolved alternate URL. - `hreflang` string — Language or locale for the alternate URL, when present. - `type` string — Alternate resource MIME type, when present. - `title` string — Alternate resource title, when present. - `headings` PageMetadataHeading[] — Page headings (h1–h6) in document order, extracted from the unfiltered document. Capped at the first 500 headings. Omitted when the page has none. - `level` integer, required — Heading level, 1–6 (from h1–h6). - `text` string, required — Heading text with whitespace collapsed, truncated to 1000 characters. - `jsonLd` object[] — JSON-LD structured data blocks parsed from the page. - `additionalMeta` object — Additional non-social meta tags not promoted to top-level metadata fields. - `url` string, required — The crawl URL fetched for this page. - `crawlDepth` integer, required — Depth relative to the start URL. 0 = start URL, 1 = one link away. - `statusCode` integer, required — HTTP status code of the response - `success` boolean, required — true if the page was fetched and parsed successfully - `metadata` object, required - `numUrls` integer, required — Total number of URLs crawled - `maxCrawlDepth` integer, required — Maximum crawl depth reached during the crawl - `numSucceeded` integer, required — Number of pages successfully crawled - `numFailed` integer, required — Number of pages that failed to crawl - `numSkipped` integer, required — Number of URLs skipped (PDFs when pdf.shouldParse=false, or URLs not matching urlRegex) - `key_metadata` KeyMetadata — Metadata about the API key used for the request. Included in every response whenever a valid API key is provided, even when the response status is not 200. - `credits_consumed` integer, required — The number of credits consumed by this request. - `credits_remaining` integer, required — The number of credits remaining for your organization after this request. ## Other responses - `400` — Bad request - Invalid URL or parameters - `401` — Unauthorized - Invalid or missing API key - `403` — Forbidden - Insufficient permissions or usage limit exceeded - `404` — Start URL returned a 404 - `408` — Request timeout - `415` — Unsupported content type - the start URL resolved to a content type that is not supported (e.g. an image, presentation, media, or archive). Supported types are HTML, XML, PDF, DOCX, DOC, XLSX, XLS, PPTX, PPT, and CSV. - `429` — Rate limit exceeded - `500` — Internal server error --- [API](https://skmtc.net/context/apis/context-dev.md) · [All operations](https://skmtc.net/context/apis/context-dev/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/context/context-dev/versions/182f532c2552/schema)