v1

latestOpenAPI 3.1.02026-07-2658119246.0 KB

Extract

Extract Async

post/v2/extract/async

Request body

auto_driver_configurationobject

Custom flow for the optimization engine: maps candidate names to the number of attempts to spend on each candidate before advancing (0 skips it). Key order defines the flow order. Providing it opts the request into 'auto' driver selection.

{"stackTrail":"paths:/v2/extract/async:post:requestBody:content:application/json:schema:allOf:properties:body","oasType":"schema","type":"unknown","description":"Request body for POST, PUT, PATCH methods","example":{"key":"value"}}

citystring

City for geolocation

consent_headerboolean

Whether to automatically handle cookie consent headers

device'desktop' | 'mobile' | 'tablet'

Device type for browser emulation

Browser driver to use

expected_status_codesinteger[]

Expected HTTP status codes for successful requests

formatsstring[]

List of acceptable response formats in order of preference

headersobject

Custom HTTP headers to include in the request

http2boolean

Whether to use HTTP/2 protocol

is_xhrboolean

Whether to emulate XMLHttpRequest behavior

markdown_backend'full_page' | 'main_content'

Selects which markdown conversion strategy to use. "full_page" converts the entire HTML page. "main_content" uses Mozilla Readability to extract the main article content before converting.

method'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE'

HTTP method for the request

os'windows' | 'mac os' | 'linux' | 'android' | 'ios'

Operating system to emulate

parseboolean

Whether to parse the response content

request_timeoutnumber

Request timeout in milliseconds

state'AL' | 'AK' | 'AS' | 'AZ' | 'AR' | 'CA' | 'CO' | 'CT' | 'DE' | 'DC' | 'FL' | 'GA' | 'GU' | 'HI' | 'ID' | 'IL' | 'IN' | 'IA' | 'KS' | 'KY' | 'LA' | 'ME' | 'MD' | 'MA' | 'MI' | 'MN' | 'MS' | 'MO' | 'MT' | 'NE' | 'NV' | 'NH' | 'NJ' | 'NM' | 'NY' | 'NC' | 'ND' | 'MP' | 'OH' | 'OK' | 'OR' | 'PA' | 'PR' | 'RI' | 'SC' | 'SD' | 'TN' | 'TX' | 'UT' | 'VT' | 'VA' | 'VI' | 'WA' | 'WV' | 'WI' | 'WY'

US state for geolocation (only valid when country is US)

tagstring

User-defined tag for request identification

urlstring required

Target URL to scrape

callback_urlstring

URL to call back when async operation completes

storage_compressboolean

Whether to compress stored data

storage_object_namestring

Custom name for the stored object

storage_typestring

Type of storage to use for results

storage_urlstring

URL for storage location

Example request

{
  "auto_driver_configuration": {
    "vx10": 2,
    "vx10-pro": 0,
    "vx6-fast": 1,
    "vx6-stealth": 1,
    "vx8": 5,
    "vx8-pro": 5
  },
  "body": {
    "key": "value"
  },
  "browser": "chrome",
  "browser_actions": [
    {
      "goto": "https://example.com/login"
    },
    {
      "wait_for_element": "#login-form"
    },
    {
      "fill": {
        "selector": "#username",
        "value": "user@example.com"
      }
    },
    {
      "fill": {
        "selector": "#password",
        "value": "password123"
      }
    },
    {
      "click": "#submit"
    },
    {
      "screenshot": {
        "full_page": true
      }
    }
  ],
  "city": "Los Angeles",
  "cookies": "sessionId=abc123; userId=user456",
  "device": "desktop",
  "driver": "vx8",
  "expected_status_codes": [
    200,
    201
  ],
  "formats": [
    "html"
  ],
  "headers": {
    "Accept-Language": "en-US",
    "User-Agent": "CustomBot/1.0"
  },
  "http2": true,
  "is_xhr": true,
  "markdown_backend": "full_page",
  "method": "GET",
  "network_capture": [
    {
      "resource_type": "document"
    }
  ],
  "os": "windows",
  "parse": true,
  "request_timeout": 30000,
  "skill": "dynamic-content",
  "state": "CA",
  "tag": "campaign-2024-q1",
  "callback_url": "https://example.com/webhook/callback",
  "storage_compress": true,
  "storage_object_name": "result-2024-01-15.json",
  "storage_type": "s3",
  "storage_url": "s3://bucket-name/path/to/object"
}

Response

Task created successfully

status'success' required

Status indicating the async task was created successfully.

Example response

{
  "task": {
    "batch_id": "4b0a90bf-c951-42e4-95b3-a95a65ba69fc",
    "created_at": "2024-01-15T10:30:00Z",
    "download_url": "https://api.webit.live/api/v2/tasks/123e4567-e89b-12d3-a456-426614174000/results",
    "error": "Connection timeout",
    "error_type": "timeout_error",
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "modified_at": "2024-01-15T10:35:00Z",
    "state": "pending",
    "status_code": 200,
    "status_url": "https://api.webit.live/api/v2/tasks/123e4567-e89b-12d3-a456-426614174000"
  }
}