Domain
robots.txt parser
Fetches and parses the robots.txt file for any domain. Returns crawl rules, sitemap URLs, and per-agent directives.
get/robots-txt
Query parameters
urlstring required
Example:https://example.com
Domain URL to fetch robots.txt from
Response
Parsed robots.txt