---
title: "Create a web crawl data source from a URL list"
method: POST
path: "/crawl-url-lists"
tags: ["crawl-url-lists"]
---

# Create a web crawl data source from a URL list

`POST /crawl-url-lists`

Creates a new web crawl data source by directly downloading a list of URLs. Unlike the standard crawl endpoint, this does not follow links or crawl pages — it downloads the specified URLs directly.

Required roles: All

## Request body

- CreateCrawlUrlListRequest
  - `name` string, required — Name of the web crawl data source
  - `target_urls` string[], required — List of URLs to download directly. Each URL must use http or https scheme.
  - `max_num_files` integer — Maximum number of files to download. Defaults to the number of target URLs if omitted.
  - `rrule` string — Recurrence rule (RFC 5545 RRULE)
  - `metadata` Metadata
    - `records` MetadataRecord[] — List of metadata records. A patch may contain up to 40 items; the merged result is capped at 20.
      - `key` string, required — Metadata key (max 20 characters)
      - `val` unknown, required
      - `type` 'string' | 'integer' | 'float' | 'date' | 'datetime' — Data type of the metadata value

## Response `200`

Successfully created web crawl data source from URL list

- Crawl
  - `id` string, required — Web crawl data source ID
  - `ingestion_setting_id` string — Web crawl ingestion setting ID
  - `name` string, required — Name of the web crawl ingestion setting
  - `start_url` string, required — Start URL of the web crawl
  - `creation_time` integer — Creation time (Unix timestamp in seconds)
  - `status` 'unknown' | 'queued' | 'not_started' | 'managed' | 'starting' | 'started' | 'success' | 'failure' | 'canceling' | 'canceled' | 'deleting' | 'delete_job_failure', required — Job status
  - `start_time` integer — Job start time (Unix timestamp in seconds)
  - `end_time` integer — Job end time (Unix timestamp in seconds)
  - `error` JobError, nullable
    - `title` string — Error title
    - `message` string — Error message
  - `metadata` Metadata
    - `records` MetadataRecord[] — List of metadata records. A patch may contain up to 40 items; the merged result is capped at 20.
      - `key` string, required — Metadata key (max 20 characters)
      - `val` unknown, required
      - `type` 'string' | 'integer' | 'float' | 'date' | 'datetime' — Data type of the metadata value

## Other responses

- `400` — Bad Request
- `401` — Unauthorized
- `500` — Internal Server Error

---

[API](https://skmtc.net/qlonolink/apis/qaip-apis.md) · [All operations](https://skmtc.net/qlonolink/apis/qaip-apis/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/qlonolink/qaip-apis/revisions/f5f5994dfe05/schema)