Sign in

v1

latestOpenAPI 3.0.32026-07-222085264.6 KB
Web APIs

Web Fetch

Fetch the HTML content of webpages given their URLs. Retrieves the page title and full HTML content for up to 10 URLs in a single request. Use this for content extraction, web content retrieval, data collection, content monitoring, and SEO analysis.

<Note> Default `rate-limit` is 10 requests per minute. Send an email to [gtm@crustdata.co](mailto:gtm@crustdata.co) to discuss higher limits if needed for your use case. </Note>
post/web/enrich/live

Headers

x-api-version'2025-11-01' required
Example:2025-11-01

API version to use. This endpoint currently requires 2025-11-01.

Request body

urlsstring[] required

List of URLs to fetch web content for. Each URL must include the protocol prefix (http:// or https://). Maximum of 10 URLs per request.

human_modeboolean

Whether to use a browser-like retrieval path when standard access is blocked.

Example request

{
  "urls": [
    "https://example.com"
  ]
}

Response

Array of fetch results, one per URL. Each result includes a success flag. Failed URLs have null fields.

successboolean nullable

Whether the content was fetched successfully for this URL.

urlstring nullable

The URL that was fetched. Null if the fetch failed.

timestampinteger nullable

Unix timestamp (seconds) of when the content was fetched. Null if the fetch failed.

titlestring nullable

The title of the fetched webpage from the HTML title tag. Null if the fetch failed.

contentstring nullable

The full HTML content of the fetched webpage. Null if the fetch failed.

Example response

[
  {
    "success": true,
    "url": "https://example.com",
    "timestamp": 1775193366,
    "title": "Example Domain"
  }
]