---
title: "Scrape a list of URLs into a knowledge base"
method: POST
path: "/knowledgebase/{id}/scrape-urls"
tags: ["Knowledge Base"]
---

# Scrape a list of URLs into a knowledge base

`POST /knowledgebase/{id}/scrape-urls`

Adds one or more URLs to a knowledge base by scraping each page's content, chunking it, and indexing for retrieval. Typical flow:

1. Discover candidate URLs (`POST /knowledgebase/get-sitemap-urls` or paste your own list).
2. Call this endpoint with the curated list — scraping runs async.
3. Poll `GET /knowledgebase/{id}/scraped-urls` for the per-URL status.

Returns `400` if your account's KB billing precheck fails (quota or plan limits). Returns `404` if the KB doesn't belong to your organization.

## Path parameters

- `id` string, required

## Request body

- object
  - `urls` string[], required

## Response `200`

Scrape job(s) queued. Poll `/scraped-urls` for per-URL status.

- object
  - `status` boolean
  - `data` object — Async job acknowledgement. Inspect `/scraped-urls` for per-URL progress.

## Other responses

- `400` — Invalid input
- `401` — Unauthorized access
- `404` — Knowledge base not found in your organization.
- `500` — Internal server error

---

[API](https://skmtc.net/smallest-inc/apis/agent-management-api.md) · [All operations](https://skmtc.net/smallest-inc/apis/agent-management-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/smallest-inc/agent-management-api/versions/6d1036a76fbc/schema)