---
title: "Create a web crawl data source"
method: POST
path: "/crawls"
tags: ["crawls"]
---

# Create a web crawl data source

`POST /crawls`

Creates a new web crawl data source and starts ingestion.

Required roles: All

## Request body

- CreateCrawlRequest
  - `name` string, required — Name of the web crawl data source
  - `start_url` string, required — Start URL of the web crawl
  - `path_filters` string[] — Path filters for crawling. The total number of characters across all elements in the array must be 2000 or fewer.
  - `content_pattern` string[] — Content patterns for filtering. The total number of characters across all elements in the array must be 2000 or fewer.
  - `max_depth` integer, required — Maximum crawl depth
  - `max_num_files` integer, required — Maximum number of files to crawl
  - `html_only` boolean — When true, only HTML files will be downloaded
  - `use_browser` boolean — Whether to use a headless browser for crawling
  - `file_extensions` string[] — File extensions to include (e.g. ".pdf", ".docx"). For supported file extensions, please refer to https://developer.qaip.com/docs/datasources#%E5%AF%BE%E5%BF%9C%E3%81%97%E3%81%A6%E3%81%84%E3%82%8B%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E5%BD%A2%E5%BC%8F
  - `rrule` string — Recurrence rule (RFC 5545 RRULE)
  - `metadata` Metadata
    - `records` MetadataRecord[] — List of metadata records. A patch may contain up to 40 items; the merged result is capped at 20.
      - `key` string, required — Metadata key (max 20 characters)
      - `val` unknown, required
      - `type` 'string' | 'integer' | 'float' | 'date' | 'datetime' — Data type of the metadata value

## Response `200`

Successfully created web crawl data source

- Crawl
  - `id` string, required — Web crawl data source ID
  - `ingestion_setting_id` string — Web crawl ingestion setting ID
  - `name` string, required — Name of the web crawl ingestion setting
  - `start_url` string, required — Start URL of the web crawl
  - `creation_time` integer — Creation time (Unix timestamp in seconds)
  - `status` 'unknown' | 'queued' | 'not_started' | 'managed' | 'starting' | 'started' | 'success' | 'failure' | 'canceling' | 'canceled' | 'deleting' | 'delete_job_failure', required — Job status
  - `start_time` integer — Job start time (Unix timestamp in seconds)
  - `end_time` integer — Job end time (Unix timestamp in seconds)
  - `error` JobError, nullable
    - `title` string — Error title
    - `message` string — Error message
  - `metadata` Metadata
    - `records` MetadataRecord[] — List of metadata records. A patch may contain up to 40 items; the merged result is capped at 20.
      - `key` string, required — Metadata key (max 20 characters)
      - `val` unknown, required
      - `type` 'string' | 'integer' | 'float' | 'date' | 'datetime' — Data type of the metadata value

## Other responses

- `400` — Bad Request
- `401` — Unauthorized
- `500` — Internal Server Error

---

[API](https://skmtc.net/qlonolink/apis/qaip-apis.md) · [All operations](https://skmtc.net/qlonolink/apis/qaip-apis/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/qlonolink/qaip-apis/revisions/f5f5994dfe05/schema)