v1

latestOpenAPI 3.0.02026-07-133020.5 KB

/scrape-js: scraping with JS rendering

Launches real Chrome browser engine, use only when /scrape endpoint features are not enough.

post/scrape-js

Request body

urlstring required

URL to scrape

waitForSelectorstring

CSS selector to wait to appear in DOM tree before page is considered as loaded.

postWaitTimeinteger

Wait for specified amount of seconds after page load (from 1 to 12s). Use this only if ScrapeNinja failed to wait for required page elements automatically.

dumpIframestring

If some particular iframe needs to be dumped, specify its name HTML value in this argument. The ScrapeNinja JS renderer will wait for <iframe name="icims_content_iframe"> to appear in DOM, then use waitForSelectorIframe CSS selector to wait for iframe DOM elements to appear inside.

waitForSelectorIframestring

If dumpIframe is activated, this property allows to wait for CSS selector inside this iframe.

extractorTargetIframeboolean

If dumpIframe is activated, this property allows to run JS extractor function against iframe HTML instead of running it against base body. This is only useful if dumpIframe is activated.

headersstring[]

Custom headers to send with the request. By default, regular Chrome browser headers are sent to the target URL.

retryNuminteger

Amount of attempts.

geostring

Geo location for basic proxy pools (you can purchase premium ScrapeNinja proxies for wider country selection and higher proxy quality). Read more about ScrapeNinja proxy setup

proxystring

Premium or your own proxy URL (overrides geo field). Read more about ScrapeNinja proxy setup

timeoutinteger

Timeout per attempt, in seconds. Each retry will take [timeout] number of seconds.

textNotExpectedstring[]

Text which will trigger a retry from another proxy address.

statusNotExpectedinteger[]

HTTP response statuses which will trigger a retry from another proxy address.

blockImagesboolean

Block images from loading. This will speed up page loading and reduce bandwidth usage.

blockMediaboolean

Block (CSS, fonts) from loading. This will speed up page loading and reduce bandwidth usage.

screenshotboolean

Take a screenshot of the page. Pass "false" to increase the speed of the request.

catchAjaxHeadersUrlMaskstring

Useful to dump some XHR response. Pass URL mask here. For example, if you need to catch all requests to https://example.com/api/data.json, pass "api/data.json" here. In response, you will get new property .info.catchedAjax with the XHR response data - { url, method, headers[], body , status, responseHeaders{} }

extractorstring

Custom JS function to extract JSON values from scraped HTML. Write&test your own extractor on https://scrapeninja.net/cheerio-sandbox/

Example request

{
  "url": "https://example.com",
  "waitForSelector": "h1.some-class",
  "postWaitTime": 5,
  "dumpIframe": "icims_content_iframe",
  "waitForSelectorIframe": ".iCIMS_PagingBatch",
  "extractorTargetIframe": true,
  "headers": [
    "X-Header: some-random-header"
  ],
  "retryNum": 1,
  "geo": "eu",
  "proxy": "http://user:pw@host:port",
  "timeout": 8,
  "textNotExpected": [
    "random-captcha-text-which-might-appear"
  ],
  "statusNotExpected": [
    403,
    429
  ],
  "blockImages": true,
  "blockMedia": true,
  "screenshot": true,
  "viewport": {
    "width": 1920,
    "height": 1080,
    "deviceScaleFactor": 3
  },
  "extractor": "function(input, cheerio) {\n    let $ = cheerio.load(input);\n return $('h1.title').text(); } "
}

Response

bodystring

HTML body of the rendered page.

Example response

{
  "info": {
    "statusCode": 200,
    "finalUrl": "https://example.com/url",
    "catchedAjax": {
      "url": "https://example.com/api/data.json",
      "method": "GET",
      "headers": [
        "content-type: xxx",
        "header2: val2"
      ],
      "body": "<html><body><h1>Hello World!</h1></body></html>",
      "bodyIframe": "<html><body><h1>Iframe content</h1></body></html>",
      "status": 200,
      "responseHeaders": {
        "content-type": "application/json"
      }
    },
    "headers": [
      "content-type: xxx",
      "header2: val2"
    ]
  },
  "body": "<html><body><h1>Hello World!</h1></body></html>"
}