Web Scraping
Crawl website sitemap
Crawls the sitemap of the given domain and returns all discovered page URLs. Supports sitemap index files (recursive), parallel fetching with concurrency control, deduplication, and filters out non-page resources (images, PDFs, etc.).
get/web/scrape/sitemap
Query parameters
domainstring required
Domain name to crawl sitemaps for (e.g., 'example.com'). The domain will be automatically normalized and validated.
maxLinksinteger
Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000.
Response
Successful response