---
title: "Crawl Sitemap"
method: GET
path: "/web/scrape/sitemap"
tags: ["Web Scraping"]
---

# Crawl Sitemap

`GET /web/scrape/sitemap`

Crawl an entire website's sitemap and return all discovered page URLs. Pass `search` to have the crawled sitemap filtered down to the pages about a phrase (for example `pricing and plans` or `api authentication docs`), most relevant first — a searched crawl scans the whole sitemap and costs 2 credits instead of 1.

## Query parameters

- `domain` string, required — Domain to build a sitemap for
- `maxLinks` integer — Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000.
- `sitemapUrl` string, uri — Optional explicit sitemap URL. When provided, exactly this sitemap is crawled instead of discovering the domain's sitemaps.
- `urlRegex` string — Optional RE2-compatible regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.
- `search` string — Optional search phrase. When provided, the crawled sitemap is filtered to the pages whose URLs are about that phrase, most relevant first, and the request costs 2 credits instead of 1.
- `headers` object — Optional outbound HTTP headers forwarded only to the target URL, sent as deep-object query params such as headers[X-Custom]=value. When provided, caching is bypassed: the result is neither read from nor written to cache.
- `timeoutMS` integer — Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).
- `zdr` 'enabled' | 'disabled' — Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.
- `tags` string[] — Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

## Response `200`

Successful response

- object
  - `success` true, required — Indicates success
  - `domain` string, required — The normalized domain that was crawled
  - `urls` string[], required — Discovered page URLs from the sitemap, up to `maxLinks`. When `search` is set these are only the matching pages, most relevant first.
  - `meta` object, required — Metadata about the sitemap crawl operation
    - `sitemapsDiscovered` integer, required — Total number of sitemap files discovered
    - `sitemapsFetched` integer, required — Number of sitemap files successfully fetched and parsed
    - `sitemapsSkipped` integer, required — Number of sitemap files skipped (due to errors, timeouts, or limits)
    - `errors` integer, required — Number of errors encountered during crawling
  - `key_metadata` KeyMetadata — Metadata about the API key used for the request. Included in every response whenever a valid API key is provided, even when the response status is not 200.
    - `credits_consumed` integer, required — The number of credits consumed by this request.
    - `credits_remaining` integer, required — The number of credits remaining for your organization after this request.

## Other responses

- `400` — Bad request - Invalid domain
- `401` — Unauthorized - Invalid or missing API key
- `403` — Forbidden - Insufficient permissions or usage limit exceeded
- `408` — Request timeout
- `429` — Rate limit exceeded
- `500` — Internal server error

---

[API](https://skmtc.net/context/apis/context-dev.md) · [All operations](https://skmtc.net/context/apis/context-dev/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/context/context-dev/versions/182f532c2552/schema)