--- title: "Retrieve crawler details" method: GET path: "/1/crawlers/{id}" tags: ["crawler"] --- # Retrieve crawler details `GET /1/crawlers/{id}` Retrieves details about the specified crawler, optionally with its configuration. ## Path parameters - `id` string, required — Universally unique identifier (UUID) of the crawler. ## Query parameters - `withConfig` boolean ## Response `200` OK - union - BaseResponse - `name` string, required — Name of the crawler. - `createdAt` string, required — Date and time when the object was created, in RFC 3339 format. - `updatedAt` string, required — Date and time when the object was updated, in RFC 3339 format. - `running` boolean, required — Whether this crawler is active. - `reindexing` boolean, required — Whether this crawler is completely reindexing your content. - `blocked` boolean, required — Whether this crawler is blocked. If `true`, you can unblock it from the [Crawler page](https://dashboard.algolia.com/crawler) in the Algolia dashboard or by [cancelling the blocking task](https://www.algolia.com/doc/rest-api/crawler/cancel-blocking-action). - `blockingError` string — Reason why the crawler is blocked. - `blockingTaskId` string — ID of the task that's blocking the crawler. - `lastReindexStartAt` string, required — Date and time when the last crawl started, in RFC 3339 format. - `lastReindexEndedAt` string, required — Date and time when the last crawl finished, in RFC 3339 format. - WithConfiguration - `name` string, required — Name of the crawler. - `createdAt` string, required — Date and time when the object was created, in RFC 3339 format. - `updatedAt` string, required — Date and time when the object was updated, in RFC 3339 format. - `running` boolean, required — Whether this crawler is active. - `reindexing` boolean, required — Whether this crawler is completely reindexing your content. - `blocked` boolean, required — Whether this crawler is blocked. If `true`, you can unblock it from the [Crawler page](https://dashboard.algolia.com/crawler) in the Algolia dashboard or by [cancelling the blocking task](https://www.algolia.com/doc/rest-api/crawler/cancel-blocking-action). - `blockingError` string — Reason why the crawler is blocked. - `blockingTaskId` string — ID of the task that's blocking the crawler. - `lastReindexStartAt` string, required — Date and time when the last crawl started, in RFC 3339 format. - `lastReindexEndedAt` string, required — Date and time when the last crawl finished, in RFC 3339 format. - `config` Configuration, required — Crawler configuration. - `actions` Action[], required — A list of actions. - `autoGenerateObjectIDs` boolean — Whether to generate an `objectID` for records that don't have one. - `cache` Cache — Whether the crawler should cache crawled pages. For more information, see [Partial crawls with caching](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#partial-crawls-with-caching). - `enabled` boolean — Whether the crawler cache is active. - `discoveryPatterns` UrlPattern[] — Which _intermediary_ web pages the crawler should visit. Use `discoveryPatterns` to define pages that should be visited _just_ for their links to other pages, _not_ their content. It functions similarly to the `pathsToMatch` action but without record extraction. - `fileTypesToMatch` FileTypes[] — File types for crawling non-HTML documents. - `hostnameAliases` HostnameAliases — Key-value pairs to replace matching hostnames found in a sitemap, on a page, in canonical links, or redirects. During a crawl, this action maps one hostname to another whenever the crawler encounters specific URLs. This helps with links to staging environments (like `dev.example.com`) or external hosting services (such as YouTube). For example, with this `hostnameAliases` mapping: { hostnameAliases: { 'dev.example.com': 'example.com' } } 1. The crawler encounters `https://dev.example.com/solutions/voice-search/`. 1. `hostnameAliases` transforms the URL to `https://example.com/solutions/voice-search/`. 1. The crawler follows the transformed URL (not the original). **`hostnameAliases` only changes URLs, not page text. In the preceding example, if the extracted text contains the string `dev.example.com`, it remains unchanged.** The crawler can discover URLs in places such as: - Crawled pages - Sitemaps - [Canonical URLs](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#canonical-urls-and-crawler-behavior) - Redirects. However, `hostnameAliases` doesn't transform URLs you explicitly set in the `startUrls` or `sitemaps` parameters, nor does it affect the `pathsToMatch` action or other configuration elements. - `indexName` string, required — Reference to the index used to store the action's extracted records. `indexName` is combined with the prefix you specified in `indexPrefix`. - `name` string — Unique identifier for the action. This option is required if `schedule` is set. - `pathAliases` PathAliases — Key-value pairs to replace matching paths with new values. It doesn't replace: - URLs in the `startUrls`, `sitemaps`, `pathsToMatch`, and other settings - Paths found in extracted text The crawl continues from the _transformed_ URLs. For example, if you create a mapping for `{ "dev.example.com": { '/foo': '/bar' } }` and the crawler encounters `https://dev.example.com/foo/hello/`, it’s transformed to `https://dev.example.com/bar/hello/`. > Compare with the `hostnameAliases` action. - `pathsToMatch` UrlPattern[] — URLs to which this action should apply. Uses [micromatch](https://github.com/micromatch/micromatch) for negation, wildcards, and more. - `recordExtractor` object, required — Function for extracting information from a crawled page and transforming it into Algolia records for indexing. The Crawler has an [editor](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#the-editor) with autocomplete and validation to help you update the `recordExtractor`. For details, see the [`recordExtractor` documentation](https://www.algolia.com/doc/tools/crawler/apis/configuration/actions/#parameter-param-recordextractor). - `__type` 'function' - `source` string — A JavaScript function (as a string) that returns one or more Algolia records for each crawled page. - `schedule` string — How often to perform a complete crawl for this action. For mopre information, consult the [`schedule` parameter documentation](https://www.algolia.com/doc/tools/crawler/apis/configuration/schedule). - `selectorsToMatch` string[] — DOM selectors for nodes that must be present on the page to be processed. If the page doesn't match any of the selectors, it's ignored. - `apiKey` string — The Algolia API key the crawler uses for indexing records. If you don't provide an API key, one will be generated by the Crawler when you create a configuration. The API key must have: - These [rights and restrictions](https://www.algolia.com/doc/guides/security/api-keys/#rights-and-restrictions): `search`, `addObject`, `deleteObject`, `deleteIndex`, `settings`, `editSettings`, `listIndexes`, `browse` - Access to the correct set of indices, based on the crawler's `indexPrefix`. For example, if the prefix is `crawler_`, the API key must have access to `crawler_*`. **Don't use your [Admin API key](https://www.algolia.com/doc/guides/security/api-keys/#predefined-api-keys)**. - `appId` string, required — Algolia application ID where the crawler creates and updates indices. - `exclusionPatterns` string[] — URLs to exclude from crawling. - `externalData` string[] — References to external data sources for enriching the extracted records. - `extraUrls` string[] — The Crawler treats `extraUrls` the same as `startUrls`. Specify `extraUrls` if you want to differentiate between URLs you manually added to fix site crawling from those you initially specified in `startUrls`. - `ignoreCanonicalTo` union - boolean — Determines if the crawler should extract records from a page with a [canonical URL](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#canonical-urls-and-crawler-behavior). If `ignoreCanonicalTo` is set to: - `true` all canonical URLs are ignored. - One or more URL patterns, the crawler will ignore the canonical URL if it matches a pattern. - string[] — Canonical URLs or URL patterns to ignore. - `ignoreNoFollowTo` boolean — Determines if the crawler should follow links with a `nofollow` directive. If `true`, the crawler will ignore the `nofollow` directive and crawl links on the page. The crawler always ignores links that don't match your [configuration settings](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#exclude-and-include-content). `ignoreNoFollowTo` applies to: - Links that are ignored because the [`robots` meta tag](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta/name#Other_metadata_names) contains `nofollow` or `none`. - Links with a [`rel` attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel) containing the `nofollow` directive. - `ignoreNoIndex` boolean — Whether to ignore the `noindex` robots meta tag. If `true`, pages with this meta tag _will_ be crawled. - `ignorePaginationAttributes` boolean — Whether the crawler should follow `rel="prev"` and `rel="next"` pagination links in the `` section of an HTML page. - If `true`, the crawler ignores the pagination links. - If `false`, the crawler follows the pagination links. - `ignoreQueryParams` string[] — Query parameters to ignore while crawling. All URLs with the matching query parameters are treated as identical. This prevents indexing URLs that just differ by their query parameters. - `ignoreRobotsTxtRules` boolean — Whether to ignore rules defined in your `robots.txt` file. - `indexPrefix` string — A prefix for all indices created by this crawler. It's combined with the `indexName` for each action to form the complete index name. - `initialIndexSettings` object — Crawler index settings. These index settings are only applied during the first crawl of an index. Any subsequent changes won't be applied to the index. Instead, make changes to your index settings in the [Algolia dashboard](https://dashboard.algolia.com/explorer/configuration). - `linkExtractor` object — Function for extracting URLs from links on crawled pages. For more information, see the [`linkExtractor` documentation](https://www.algolia.com/doc/tools/crawler/apis/configuration/link-extractor). - `__type` 'function' - `source` string - `login` union — Authorization method and credentials for crawling protected content. The Crawler supports these authentication methods: - **Basic authentication**. The Crawler obtains a session cookie from the login page. - **OAuth 2.0 authentication** (`oauthRequest`). The Crawler uses OAuth 2.0 client credentials to obtain an access token for authentication. **Basic authentication** The Crawler extracts the `Set-Cookie` response header from the login page, stores that cookie, and sends it in the `Cookie` header when crawling all pages defined in the configuration. This cookie is retrieved only at the start of each full crawl. If it expires, it isn't automatically renewed. The Crawler can obtain the session cookie in one of two ways: - **HTTP request authentication** (`fetchRequest`). The Crawler sends a direct request with your credentials to the login endpoint, similar to a `curl` command. - **Browser-based authentication** (`browserRequest`). The Crawler emulates a web browser by loading the login page, entering the credentials, and submitting the login form as a real user would. **OAuth 2.0** The crawler supports [OAuth 2.0 client credentials grant flow](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4): 1. It performs an access token request with the provided credentials 1. Stores the fetched token in an `Authorization` header 1. Sends the token when crawling site pages. This token is only fetched at the beginning of each complete crawl. If it expires, it isn't automatically renewed. Client authentication passes the credentials (`client_id` and `client_secret`) [in the request body](https://datatracker.ietf.org/doc/html/rfc6749#section-2.3.1). The [Azure AD v1.0](https://learn.microsoft.com/en-us/previous-versions/azure/active-directory/azuread-dev/v1-oauth2-client-creds-grant-flow) provider is supported. - FetchRequest — Information for making a HTTP request for authorization. - `url` string, required — URL with your login form. - `requestOptions` LoginRequestOptions — Options for the HTTP request for logging in. - `method` string — HTTP method for sending the request. - `headers` Headers — Headers to add to all requests. - … - `body` string — Form content. - `timeout` integer — Timeout for the request. - BrowserRequest — Information for using a web browser for authorization. The browser loads a login page and enters the provided credentials. - `url` string, required — URL of your login page. The crawler looks for an input matching the selector `input[type=text]` or `input[type=email]` for the username and `input[type=password]` for the password. - `username` string, required — Username for signing in. - `password` string, required — Password for signing in. - `waitTime` WaitTime — Timeout for the HTTP request. - `min` integer — Minimum waiting time in milliseconds. - `max` integer — Maximum waiting time in milliseconds. - OauthRequest — Authorization information for using the [OAuth 2.0 client credentials](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4) authorization grant. OAuth authorization is supported for [Azure Active Directory version 1](https://learn.microsoft.com/en-us/previous-versions/azure/active-directory/azuread-dev/v1-oauth2-client-creds-grant-flow) as provider. - `accessTokenRequest` AccessTokenRequest, required — Parameters required to make the [access token request](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.2). - `url` string, required — URL for the access token endpoint. - `grantType` 'client_credentials', required — OAuth 2.0 grant type. - `clientId` string, required — [Client identifier](https://datatracker.ietf.org/doc/html/rfc6749#section-2.2). - `clientSecret` string, required — Client secret. - `scope` string — [Access token scope](https://datatracker.ietf.org/doc/html/rfc6749#section-3.3). - `extraParameters` ExtraParameters — Extra parameters for the authorization request. - … - `maxDepth` integer — Determines the maximum path depth of crawled URLs. Path depth is calculated based on the number of slash characters (`/`) after the domain (starting at 1). For example: - **1** `http://example.com` - **1** `http://example.com/` - **1** `http://example.com/foo` - **2** `http://example.com/foo/` - **2** `http://example.com/foo/bar` - **3** `http://example.com/foo/bar/` **URLs added with `startUrls` and `sitemaps` aren't checked for `maxDepth`.**. - `maxUrls` integer — Limits the number of URLs your crawler processes. Change it to a low value, such as 100, for short crawling tests. Change it to a higher explicit value for full crawls to prevent it from getting "lost" in complex site structures. Because the Crawler works on many pages simultaneously, `maxUrls` doesn't guarantee finding the same pages each time it runs. - `rateLimit` integer, required — Determines the number of concurrent tasks per second that can run for this configuration. A higher rate limit means more crawls per second. Algolia prevents system overload by ensuring the number of URLs added in the last second and the number of URLs being processed is less than the rate limit: ``` max(new_urls_added, active_urls_processing) <= rateLimit ``` Start with a low value (for example, 2) and increase it if you need faster crawling. A high `rateLimit` can significantly increase bandwidth cost and server resource consumption. The number of pages processed per second depends on the average time it takes to fetch, process, and upload a URL. For a given `rateLimit`, if fetching, processing, and uploading URLs takes (on average): - Less than a second, your crawler processes up to `rateLimit` pages per second. - Four seconds, your crawler processes up to `rateLimit / 4` pages per second. In the latter case, increasing `rateLimit` improves performance up to a point. If the processing time remains at four seconds, increasing `rateLimit` won't increase the number of pages processed per second. - `renderJavaScript` union — If `true`, use a Chrome headless browser to crawl pages. Because crawling JavaScript-based web pages is slower than crawling regular HTML pages, you can apply this setting to a specific list of pages. Use [micromatch](https://github.com/micromatch/micromatch) to define URL patterns, including negations and wildcards. - boolean — Whether to render all pages. - string[] — URLs or URL patterns to render. - object — Configuration for rendering HTML. - `enabled` boolean, required — Whether to enable JavaScript rendering. - `patterns` string[], required — URLs or URL patterns to render. - `adBlock` boolean — Whether to use the Crawler's ad blocker. It blocks most ads and tracking scripts but can break some sites. - `waitTime` WaitTime — Timeout for the HTTP request. - `min` integer — Minimum waiting time in milliseconds. - `max` integer — Maximum waiting time in milliseconds. - `requestOptions` RequestOptions — Lets you add options to HTTP requests made by the crawler. - `proxy` string — Proxy for all crawler requests. - `timeout` integer — Timeout in milliseconds for the crawl. - `retries` integer — Maximum number of retries to crawl one URL. - `headers` Headers — Headers to add to all requests. - `Accept-Language` string — Preferred natural language and locale. - `Authorization` string — Basic authentication header. - `Cookie` string — Cookie. The header will be replaced by the cookie retrieved when logging in. - `safetyChecks` SafetyChecks — Checks to ensure the crawl was successful. For more information, see the [Safety checks](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration/#safety-checks) documentation. - `beforeIndexPublishing` BeforeIndexPublishing — Checks triggered after the crawl finishes but before the records are added to the Algolia index. - `maxLostRecordsPercentage` integer — Maximum difference in percent between the numbers of records between crawls. - `maxFailedUrls` integer — Stops the crawler if a specified number of pages fail to crawl. - `saveBackup` boolean — Whether to back up your index before the crawler overwrites it with new records. - `schedule` string — Schedule for running the crawl. Instead of manually starting a crawl each time, you can set up a schedule for automatic crawls. [Use the visual UI](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration-visual) or add the `schedule` parameter to [your configuration](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-configuration). `schedule` uses [Later.js syntax](https://bunkat.github.io/later) to specify when to crawl your site. Here are some key things to keep in mind when using `Later.js` syntax with the Crawler: - The interval between two scheduled crawls must be at least 24 hours. - To crawl daily, use "every 1 day" instead of "everyday" or "every day". - If you don't specify a time, the crawl can happen any time during the scheduled day. - Specify times for the UTC (GMT+0) timezone - Include minutes when specifying a time. For example, "at 3:00 pm" instead of "at 3pm". - Use "at 12:00 am" to specify midnight, not "at 00:00 am". - `sitemaps` string[] — Sitemaps with URLs from where to start crawling. - `startUrls` string[] — URLs from where to start crawling. ## Other responses - `400` — Invalid request. - `401` — Authorization information is missing or invalid. - `403` — The user doesn't have enough rights on the specified Crawler, or it doesn't exists. --- [API](https://skmtc.net/algolia/apis/crawler-api.md) · [All operations](https://skmtc.net/algolia/apis/crawler-api/llms.txt) · [OpenAPI document](https://skmtc-service-staging.skmtc.workers.dev/v1/apis/algolia/crawler-api/versions/7f1df3bfe8bd/schema)