Sign in

v49

latestOpenAPI 3.0.2raw.githubusercontent.com2026-02-0620117129.6 KB
crawler

Update crawler configuration

Updates the configuration of the specified crawler. Every time you update the configuration, a new version is created.

patch/1/crawlers/{id}/config

Path parameters

idstring required

Universally unique identifier (UUID) of the crawler.

Example:e0f6db8a-24f5-4092-83a4-1b2c6cb6d809

Crawler ID.

Request body

apiKeystring

The Algolia API key the crawler uses for indexing records. If you don't provide an API key, one will be generated by the Crawler when you create a configuration.

The API key must have:

  • These rights and restrictions: search, addObject, deleteObject, deleteIndex, settings, editSettings, listIndexes, browse
  • Access to the correct set of indices, based on the crawler's indexPrefix. For example, if the prefix is crawler_, the API key must have access to crawler_*.

Don't use your Admin API key.

appIdstring required

Algolia application ID where the crawler creates and updates indices.

exclusionPatternsstring[]

URLs to exclude from crawling.

externalDatastring[]

References to external data sources for enriching the extracted records.

extraUrlsstring[]

The Crawler treats extraUrls the same as startUrls. Specify extraUrls if you want to differentiate between URLs you manually added to fix site crawling from those you initially specified in startUrls.

ignoreNoFollowToboolean

Determines if the crawler should follow links with a nofollow directive. If true, the crawler will ignore the nofollow directive and crawl links on the page.

The crawler always ignores links that don't match your configuration settings. ignoreNoFollowTo applies to:

  • Links that are ignored because the robots meta tag contains nofollow or none.
  • Links with a rel attribute containing the nofollow directive.
ignoreNoIndexboolean

Whether to ignore the noindex robots meta tag. If true, pages with this meta tag will be crawled.

ignorePaginationAttributesboolean

Whether the crawler should follow rel="prev" and rel="next" pagination links in the <head> section of an HTML page.

  • If true, the crawler ignores the pagination links.
  • If false, the crawler follows the pagination links.
ignoreQueryParamsstring[]

Query parameters to ignore while crawling.

All URLs with the matching query parameters are treated as identical. This prevents indexing URLs that just differ by their query parameters.

ignoreRobotsTxtRulesboolean

Whether to ignore rules defined in your robots.txt file.

indexPrefixstring

A prefix for all indices created by this crawler. It's combined with the indexName for each action to form the complete index name.

initialIndexSettingsobject

Crawler index settings.

These index settings are only applied during the first crawl of an index.

Any subsequent changes won't be applied to the index. Instead, make changes to your index settings in the Algolia dashboard.

maxDepthinteger

Determines the maximum path depth of crawled URLs.

Path depth is calculated based on the number of slash characters (/) after the domain (starting at 1). For example:

  • 1 http://example.com
  • 1 http://example.com/
  • 1 http://example.com/foo
  • 2 http://example.com/foo/
  • 2 http://example.com/foo/bar
  • 3 http://example.com/foo/bar/

URLs added with startUrls and sitemaps aren't checked for maxDepth..

maxUrlsinteger

Limits the number of URLs your crawler processes.

Change it to a low value, such as 100, for short crawling tests. Change it to a higher explicit value for full crawls to prevent it from getting "lost" in complex site structures. Because the Crawler works on many pages simultaneously, maxUrls doesn't guarantee finding the same pages each time it runs.

rateLimitinteger required

Determines the number of concurrent tasks per second that can run for this configuration.

A higher rate limit means more crawls per second. Algolia prevents system overload by ensuring the number of URLs added in the last second and the number of URLs being processed is less than the rate limit:

max(new_urls_added, active_urls_processing) <= rateLimit

Start with a low value (for example, 2) and increase it if you need faster crawling. A high rateLimit can significantly increase bandwidth cost and server resource consumption.

The number of pages processed per second depends on the average time it takes to fetch, process, and upload a URL. For a given rateLimit, if fetching, processing, and uploading URLs takes (on average):

  • Less than a second, your crawler processes up to rateLimit pages per second.
  • Four seconds, your crawler processes up to rateLimit / 4 pages per second.

In the latter case, increasing rateLimit improves performance up to a point. If the processing time remains at four seconds, increasing rateLimit won't increase the number of pages processed per second.

saveBackupboolean

Whether to back up your index before the crawler overwrites it with new records.

schedulestring

Schedule for running the crawl.

Instead of manually starting a crawl each time, you can set up a schedule for automatic crawls. Use the visual UI or add the schedule parameter to your configuration.

schedule uses Later.js syntax to specify when to crawl your site. Here are some key things to keep in mind when using Later.js syntax with the Crawler:

  • The interval between two scheduled crawls must be at least 24 hours.
  • To crawl daily, use "every 1 day" instead of "everyday" or "every day".
  • If you don't specify a time, the crawl can happen any time during the scheduled day.
  • Specify times for the UTC (GMT+0) timezone
  • Include minutes when specifying a time. For example, "at 3:00 pm" instead of "at 3pm".
  • Use "at 12:00 am" to specify midnight, not "at 00:00 am".
sitemapsstring[]

Sitemaps with URLs from where to start crawling.

startUrlsstring[]

URLs from where to start crawling.

Example request

{
  "actions": [
    {
      "discoveryPatterns": [
        "https://www.algolia.com/**"
      ],
      "fileTypesToMatch": [
        "html",
        "pdf"
      ],
      "hostnameAliases": {
        "dev.example.com": "example.com"
      },
      "indexName": "algolia_website",
      "pathAliases": {
        "example.com": {
          "/foo": "/bar"
        }
      },
      "pathsToMatch": [
        "https://www.algolia.com/**"
      ],
      "selectorsToMatch": [
        ".products",
        "!.featured"
      ]
    }
  ],
  "exclusionPatterns": [
    "https://www.example.com/excluded",
    "!https://www.example.com/this-one-url",
    "https://www.example.com/exclude/**"
  ],
  "externalData": [
    "testCSV"
  ],
  "ignoreQueryParams": [
    "ref",
    "utm_*"
  ],
  "indexPrefix": "crawler_",
  "linkExtractor": {
    "source": "({ $, url, defaultExtractor }) => {\n  if (/example.com\\/doc\\//.test(url.href)) {\n    // For all pages under `/doc`, only extract the first found URL.\n    return defaultExtractor().slice(0, 1)\n  }\n  // For all other pages, use the default.\n  return defaultExtractor()\n}\n"
  },
  "login": {
    "url": "https://example.com/secure/login-with-post",
    "requestOptions": {
      "method": "POST",
      "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
      },
      "body": "id=my-id&password=my-password",
      "timeout": 5000
    }
  },
  "maxDepth": 5,
  "maxUrls": 250,
  "rateLimit": 4,
  "renderJavaScript": {
    "enabled": true,
    "patterns": [
      "http://www.mysite.com/dynamic-pages/**"
    ],
    "waitTime": {
      "min": 7000,
      "max": 15000
    }
  },
  "requestOptions": {
    "headers": {
      "Accept-Language": "fr-FR",
      "Authorization": "Bearer Aerehdf==",
      "Cookie": "session=1234"
    }
  },
  "schedule": "every weekday at 12:00 pm",
  "sitemaps": [
    "https://example.com/sitemap.xyz"
  ],
  "startUrls": [
    "https://www.example.com"
  ]
}

Response

OK

taskIdstring required

Universally unique identifier (UUID) of the task.

Example response

{
  "taskId": "98458796-b7bb-4703-8b1b-785c1080b110"
}