v1

latestOpenAPI 3.1.02026-07-136138.4 KB

Distill

Distill Single Page

Convert a web page into clean, LLM-ready Markdown format.

Use Cases:

Prepare web content for RAG (Retrieval-Augmented Generation)
Extract article content for AI processing
Convert documentation pages to markdown
Process dynamic web applications

What's Included:

Clean markdown content with preserved structure
Automatic removal of ads, navigation, and boilerplate
Metadata extraction (title, description, language)
JavaScript rendering for dynamic content
Automatic handling of anti-bot measures

Output Format: Returns markdown optimized for LLM consumption with minimal noise and maximum signal.

post/distill

Request body

urlstring required

The URL of the web page to distill

timeoutnumber

Request timeout in milliseconds (default: 30000, max: 60000)

waitFornumber

Time to wait (in milliseconds) after page load for dynamic content to render before extracting content

includeTagsstring[]

Only include content from these HTML tags (e.g., ['article', 'main', 'div.content'])

excludeTagsstring[]

Exclude content from these HTML tags (e.g., ['nav', 'footer', 'aside'])

headersobject

Custom HTTP headers to send with the request

Example request

{
  "url": "https://example.com/article",
  "timeout": 30000,
  "waitFor": 2000,
  "includeTags": [
    "article",
    "main"
  ],
  "excludeTags": [
    "nav",
    "footer",
    "aside"
  ],
  "headers": {
    "User-Agent": "MyBot/1.0",
    "Accept-Language": "en-US"
  }
}

Response

Success response

successboolean

Example response

{
  "success": true,
  "data": {
    "url": "https://example.com/article",
    "markdown": "# Understanding Web Distillation\n\nWeb distillation is the process of converting raw HTML into clean, structured content...\n\n## Key Benefits\n\n- **LLM-Ready Format**: Optimized for AI consumption\n- **Noise Removal**: Automatic filtering of ads and boilerplate\n- **Structure Preservation**: Maintains semantic hierarchy\n\n## Use Cases\n\nWeb distillation is ideal for:\n\n1. Building RAG systems\n2. Content analysis\n3. Knowledge base creation",
    "html": "<article><h1>Understanding Web Distillation</h1>...</article>",
    "metadata": {
      "title": "Understanding Web Distillation | Tech Blog",
      "description": "Learn how web distillation converts raw HTML into clean, LLM-ready content for AI applications.",
      "language": "en",
      "author": "John Doe",
      "publishedDate": "2024-01-15T10:00:00Z",
      "image": "https://example.com/images/featured.jpg",
      "sourceURL": "https://example.com/article",
      "statusCode": 200,
      "contentLength": 4523
    },
    "links": [
      {
        "text": "Related Article",
        "href": "https://example.com/related"
      }
    ]
  }
}

All 6 operations