---
title: "Ingest Web"
method: POST
path: "/api/kb/ingest-web"
tags: ["kb"]
---

# Ingest Web

`POST /api/kb/ingest-web`

Ingest website content into the knowledge base.

Args:
    collection: Target collection name
    start_url: Starting URL for crawling
    max_pages: Maximum number of pages to crawl
    max_depth: Maximum crawl depth
    url_patterns: Comma-separated URL match patterns (regex)
    exclude_patterns: Comma-separated exclusion patterns (regex)
    same_domain_only: Only crawl same domain
    content_selector: CSS selector for main content area
    remove_selectors: Comma-separated CSS selectors to remove
    concurrent_requests: Number of concurrent requests
    request_delay: Delay between requests in seconds
    timeout: Request timeout in seconds
    respect_robots_txt: Respect robots.txt rules
    parse_method: Parser for document ingestion
    chunk_strategy: Chunking strategy
    chunk_size: Chunk size in characters
    chunk_overlap: Chunk overlap in characters
    embedding_model_id: Embedding model ID
    embedding_batch_size: Batch size for embedding
    max_retries: Maximum retry attempts
    retry_delay: Delay between retries

## Response `200`

Successful Response

- WebIngestionResult — Result of website crawling and knowledge base ingestion. This model provides comprehensive statistics about the crawling and ingestion process, including success/failure counts and timing.
  - `status` string, required — Overall status: success|error|partial
  - `collection` string, required — Target collection name
  - `total_urls_found` integer, required — Total number of unique URLs discovered
  - `pages_crawled` integer, required — Number of successfully crawled pages
  - `pages_failed` integer, required — Number of pages that failed to crawl
  - `documents_created` integer, required — Number of documents created in collection
  - `chunks_created` integer, required — Total number of chunks created
  - `embeddings_created` integer, required — Total number of embeddings generated
  - `crawled_urls` string[] — List of successfully crawled URLs
  - `failed_urls` object — Map of failed URLs to error messages
  - `message` string, required — Human-readable summary message
  - `warnings` string[] — Non-critical warnings
  - `side_effects_may_remain` boolean — Whether rollback failed after an ingestion error and persisted side effects may still exist
  - `elapsed_time_ms` integer, required — Total elapsed time in milliseconds

## Other responses

- `422` — Validation Error

---

[API](https://skmtc.net/xorbitsai/apis/xagent.md) · [All operations](https://skmtc.net/xorbitsai/apis/xagent/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/xorbitsai/xagent/revisions/33e4ba4936ad/schema)