v1

latestOpenAPI 3.0.3BSL2026-07-17133415718.8 KB

Crawl

Get all crawl requests for a dataset

This endpoint is used to get all crawl requests for a dataset.

get/api/crawl

Query parameters

pageinteger nullable

The page number to retrieve

limitinteger nullable

The number of items to retrieve per page

Headers

TR-Datasetstring uuid required

The dataset id to use for the request

Response

Crawl requests retrieved successfully

attempt_numberinteger required

crawl_type'firecrawl' | 'openapi' | 'shopify' | 'youtube' required

created_atstring date-time required

dataset_idstring uuid required

idstring uuid required

intervalstring nullable

next_crawl_atstring date-time nullable

scrape_idstring uuid required

urlstring required

Example response

[
  {
    "crawl_options": {
      "crawl_options": {
        "allow_external_links": false,
        "boost_titles": true,
        "exclude_tags": [
          "#ad",
          "#footer",
          "header",
          "head",
          "navbar",
          "footer",
          "aside",
          "nav",
          "form"
        ],
        "heading_remove_strings": [
          "Advertisement",
          "Sponsored"
        ],
        "ignore_sitemap": true,
        "include_tags": [],
        "interval": "daily",
        "limit": 50,
        "site_url": "nedzo.ai"
      }
    }
  }
]