Sign in

v1

latestOpenAPI 3.0.02026-07-263039205.4 KB

Manage a Crawl Job

Pause, delete, restart, or view the status of a crawl job.

get/crawl

Query parameters

tokenstring required

Your Diffbot token

namestring

Job name as defined when the crawl job was created.

roundStartstring

Pass roundStart=1 to force the start of a new crawl "round" (manually repeat the crawl). If onlyProcessIfNew is set to 1 (default), only newly-created pages will be processed.

pausestring

Pass pause=1 to pause a crawl job. Pass pause=0 to resume a paused job.

restartstring

Pass restart=1 to restart a crawl job. This will erase all processed data and re-process all of the submitted URLs.

deletestring

Pass delete=1 to delete a job, and all associated data, completely.

Response

200

OR
OR
OR

Example response

{
  "response": "Successfully added urls for spidering.",
  "jobs": [
    {
      "jobStatus": {
        "message": "Job is initializing."
      },
      "maxHops": -1,
      "downloadJson": "...json",
      "maxRounds": -1,
      "type": "crawl",
      "apiUrl": "https://api.diffbot.com/v3/analyze",
      "useCanonical": 1,
      "jobCreationTimeUTC": 1649950325,
      "downloadUrls": "...csv",
      "obeyRobots": 1,
      "seedRecrawlFrequency": -1,
      "crawlDelay": -1,
      "currentTime": 1649950325,
      "currentTimeUTC": 1649950325,
      "name": "crawlTest"
    }
  ]
}