v38

latestOpenAPI 3.1.0raw.githubusercontent.com2026-06-1793196344.2 KB

Integrations

Integrations

Crawl a website for indexed search

Recursively crawl a website to make it available for indexed search.

get/integrations/web_crawler/index

Query parameters

urlstring required

The base URL of the website to crawl

The base URL of the website to crawl

max_depthinteger

Maximum depth of links to follow during crawling

Maximum depth of links to follow during crawling

limitinteger

Maximum number of pages to crawl in total

Maximum number of pages to crawl in total

Response

Successful Response

source'reddit' | 'notion' | 'slack' | 'google_calendar' | 'google_mail' | 'box' | 'dropbox' | 'github' | 'google_drive' | 'vault' | 'web_crawler' | 'trace' | 'microsoft_teams' | 'gmail_actions' | 'granola' | 'fathom' | 'fireflies' | 'linear' | 'hubspot' | 'salesforce' | 'coda' | 'lightfield' | 'gong' required

resource_idstring required

status'pending' | 'processing' | 'completed' | 'failed' | 'pending_review' | 'skipped' required