---
title: "Create a Crawl"
method: POST
path: "/crawl"
---

# Create a Crawl

`POST /crawl`

Create and start a job to spider and extract pages through a site.

## Response `200`

200

- object
  - `response` string
  - `jobs` object[]
    - `maxToProcessPerSubdomain` integer
    - `jobStatus` object
      - `message` string
      - `status` integer
    - `maxHops` integer
    - `onlyProcessIfNew` integer
    - `downloadJson` string
    - `urlProcessPattern` string
    - `jobCompletionTimeUTC` integer
    - `maxRounds` integer
    - `restrictDomain` integer
    - `type` string
    - `pageCrawlSuccessesThisRound` integer
    - `urlCrawlRegEx` string
    - `pageProcessPattern` string
    - `apiUrl` string
    - `useCanonical` integer
    - `maxToCrawlPerSubdomain` integer
    - `jobCreationTimeUTC` integer
    - `repeat` integer
    - `downloadUrls` string
    - `obeyRobots` integer
    - `roundsCompleted` integer
    - `pageCrawlAttempts` integer
    - `notifyWebhook` string
    - `pageProcessSuccessesThisRound` integer
    - `customHeaders` object
    - `objectsFound` integer
    - `roundStartTime` integer
    - `urlCrawlPattern` string
    - `maxToProcess` integer
    - `seedRecrawlFrequency` integer
    - `seeds` string
    - `urlProcessRegEx` string
    - `pageProcessSuccesses` integer
    - `urlsHarvested` integer
    - `crawlDelay` number
    - `currentTime` integer
    - `useProxies` integer
    - `maxToCrawl` integer
    - `sentJobDoneNotification` integer
    - `currentTimeUTC` integer
    - `restrictSubdomain` integer
    - `name` string
    - `notifyEmail` string
    - `pageCrawlSuccesses` integer
    - `pageProcessAttempts` integer

## Other responses

- `400` — 400
- `505` — 505

---

[API](https://skmtc.net/diffbot/apis/web-search.md) · [All operations](https://skmtc.net/diffbot/apis/web-search/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/diffbot/web-search/versions/8b88d7dabe63/schema)