Knowledge Base
Scrape a list of URLs into a knowledge base
Adds one or more URLs to a knowledge base by scraping each page's content, chunking it, and indexing for retrieval. Typical flow:
- Discover candidate URLs (POST /knowledgebase/get-sitemap-urls or paste your own list).
- Call this endpoint with the curated list — scraping runs async.
- Poll GET /knowledgebase/{id}/scraped-urls for the per-URL status.
Returns 400 if your account's KB billing precheck fails (quota or plan limits). Returns 404 if the KB doesn't belong to your organization.
post/knowledgebase/{id}/scrape-urls
Path parameters
idstring required
24-char hex ObjectId of the target knowledge base.
Request body
Example request
{
"urls": [
"https://example.com/pricing",
"https://example.com/faq"
]
}Response
Scrape job(s) queued. Poll /scraped-urls for per-URL status.
Example response
{
"status": true
}