Ingest Web

Ingest website content into the knowledge base.

Args: collection: Target collection name start_url: Starting URL for crawling max_pages: Maximum number of pages to crawl max_depth: Maximum crawl depth url_patterns: Comma-separated URL match patterns (regex) exclude_patterns: Comma-separated exclusion patterns (regex) same_domain_only: Only crawl same domain content_selector: CSS selector for main content area remove_selectors: Comma-separated CSS selectors to remove concurrent_requests: Number of concurrent requests request_delay: Delay between requests in seconds timeout: Request timeout in seconds respect_robots_txt: Respect robots.txt rules parse_method: Parser for document ingestion chunk_strategy: Chunking strategy chunk_size: Chunk size in characters chunk_overlap: Chunk overlap in characters embedding_model_id: Embedding model ID embedding_batch_size: Batch size for embedding max_retries: Maximum retry attempts retry_delay: Delay between retries