---
title: "Update a crawl request"
method: PUT
path: "/api/crawl"
tags: ["Crawl"]
---

# Update a crawl request

`PUT /api/crawl`

This endpoint is used to update an existing crawl request for a dataset. The request payload should contain the crawl id and the crawl options to update for the crawl.

## Headers

- `TR-Dataset` string, uuid, required

## Request body

- UpdateCrawlReqPayload
  - `crawl_id` string, uuid, required — Crawl ID to update
  - `crawl_options` CrawlOptions, required — Options for setting up the crawl which will populate the dataset.
    - `add_chunks_to_dataset` boolean, nullable — Add chunks to the dataset that the crawl is created for, defaults to true
    - `allow_external_links` boolean, nullable — Option for allowing the crawl to follow links to external websites.
    - `body_remove_strings` string[], nullable — Text strings to remove from body when creating chunks for each page
    - `boost_titles` boolean, nullable — Boost titles such that keyword matches in titles are prioritized in search results. Strongly recommended to leave this on. Defaults to true.
    - `exclude_paths` string[], nullable — URL Patterns to exclude from the crawl
    - `exclude_tags` string[], nullable — Specify the HTML tags, classes and ids to exclude from the response.
    - `heading_remove_strings` string[], nullable — Text strings to remove from headings when creating chunks for each page
    - `ignore_sitemap` boolean, nullable — Ignore the website sitemap when crawling, defaults to true.
    - `include_paths` string[], nullable — URL Patterns to include in the crawl
    - `include_tags` string[], nullable — Specify the HTML tags, classes and ids to include in the response.
    - `interval` 'daily' | 'weekly' | 'monthly' — Interval at which specified site should be re-scraped
    - `limit` integer, nullable — How many pages to crawl, defaults to 1000
    - `scrape_options` union — Options for including an openapi spec or shopify settigns
      - object — Options for including an openapi spec in the crawl
        - `openapi_schema_url` string, required — OpenAPI json schema to be processed alongside the site crawl
        - `openapi_tag` string, required — Tag to look for to determine if a page should create an openapi route chunk instead of chunks from heading-split of the HTML
        - `type` 'openapi', required
      - object — Options for Crawling Shopify
        - `group_variants` boolean, nullable — This option will ingest all variants as individual chunks and place them in groups by product id. Turning this off will only scrape 1 variant per product. default: true
        - `tag_regexes` string[], nullable
        - `type` 'shopify', required
      - object — Options for Crawling Youtube
        - `type` 'youtube', required
    - `site_url` string, nullable — The URL to crawl
    - `tags` string[], nullable — Tags to add to the crawl
    - `webhook_metadata` unknown
    - `webhook_urls` string[], nullable — Host to call back on the webhook for each successful page scrape

## Response `200`

Crawl updated successfully

- CrawlRequest
  - `attempt_number` integer, required
  - `crawl_options` CrawlOptions, required — Options for setting up the crawl which will populate the dataset.
    - `add_chunks_to_dataset` boolean, nullable — Add chunks to the dataset that the crawl is created for, defaults to true
    - `allow_external_links` boolean, nullable — Option for allowing the crawl to follow links to external websites.
    - `body_remove_strings` string[], nullable — Text strings to remove from body when creating chunks for each page
    - `boost_titles` boolean, nullable — Boost titles such that keyword matches in titles are prioritized in search results. Strongly recommended to leave this on. Defaults to true.
    - `exclude_paths` string[], nullable — URL Patterns to exclude from the crawl
    - `exclude_tags` string[], nullable — Specify the HTML tags, classes and ids to exclude from the response.
    - `heading_remove_strings` string[], nullable — Text strings to remove from headings when creating chunks for each page
    - `ignore_sitemap` boolean, nullable — Ignore the website sitemap when crawling, defaults to true.
    - `include_paths` string[], nullable — URL Patterns to include in the crawl
    - `include_tags` string[], nullable — Specify the HTML tags, classes and ids to include in the response.
    - `interval` 'daily' | 'weekly' | 'monthly' — Interval at which specified site should be re-scraped
    - `limit` integer, nullable — How many pages to crawl, defaults to 1000
    - `scrape_options` union — Options for including an openapi spec or shopify settigns
      - object — Options for including an openapi spec in the crawl
        - `openapi_schema_url` string, required — OpenAPI json schema to be processed alongside the site crawl
        - `openapi_tag` string, required — Tag to look for to determine if a page should create an openapi route chunk instead of chunks from heading-split of the HTML
        - `type` 'openapi', required
      - object — Options for Crawling Shopify
        - `group_variants` boolean, nullable — This option will ingest all variants as individual chunks and place them in groups by product id. Turning this off will only scrape 1 variant per product. default: true
        - `tag_regexes` string[], nullable
        - `type` 'shopify', required
      - object — Options for Crawling Youtube
        - `type` 'youtube', required
    - `site_url` string, nullable — The URL to crawl
    - `tags` string[], nullable — Tags to add to the crawl
    - `webhook_metadata` unknown
    - `webhook_urls` string[], nullable — Host to call back on the webhook for each successful page scrape
  - `crawl_type` 'firecrawl' | 'openapi' | 'shopify' | 'youtube', required
  - `created_at` string, date-time, required
  - `dataset_id` string, uuid, required
  - `id` string, uuid, required
  - `interval` string, nullable
  - `next_crawl_at` string, date-time, nullable
  - `scrape_id` string, uuid, required
  - `status` union, required
    - 'Pending'
    - object
      - `Processing` integer, required
    - 'Completed'
    - 'Failed'
  - `url` string, required

## Other responses

- `400` — Service error relating to updating the dataset

---

[API](https://skmtc.net/devflowinc/apis/trieve-api.md) · [All operations](https://skmtc.net/devflowinc/apis/trieve-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/devflowinc/trieve-api/revisions/84583e7c9fc1/schema)