Sitemap Extractor + robots.txt & llms.txt Checker for AI

Sitemap Extractor + robots.txt and llms.txt Checker for AI Crawlers

Give it domains and get every URL from their sitemaps (indexes, gzip, nested), the robots.txt verdict for each AI and search crawler (GPTBot, ClaudeBot, Google-Extended, PerplexityBot...) with the rule that decided, and the site's llms.txt, as one JSON record per site. $0.002 per site. Optional: the site's page metadata and brand kit (logo, favicon, Open Graph, social profiles, contact emails and phones).

For SEO and AI-visibility audits, crawler and scraper builders who need a URL list without crawling, and AI agents (Claude, ChatGPT, Cursor and others through the Apify MCP server) that must check whether they may read a site and where its docs are.

Use cases

• Sitemap URL extraction: a site's full URL inventory with lastmod, filtered by path prefix or regex.
• AI crawler access audit: which AI bots a site blocks or allows, and the exact robots.txt line that decides.
• llms.txt check: whether a site publishes llms.txt / llms-full.txt, parsed into sections and links.
• Crawler seeding and change monitoring: feed the URL list to a scraper; compare lastmod between runs.
• "May I fetch this URL?": check mode tests any URLs against robots.txt for the user agents you name.
• Website metadata and brand kit: with extractMetadata, each site's title, description, logo, favicon, theme colour, Open Graph tags, JSON-LD Organization, social profile links and published emails and phones, for lead enrichment, company directories, link previews and CRM logos.

Quick start

This input (the Actor's example) analyzes two sites in about 5 seconds for about $0.006:

{"sites": ["nytimes.com", "developers.cloudflare.com"], "maxUrls": 1000}

One dataset item per site (abridged, real output from 2026-10-24):

{
  "input": "nytimes.com",
  "site": "https://nytimes.com",
  "status": "ok",
  "crawlerAccess": {
    "GPTBot": {"allowed": false, "group": "GPTBot", "rule": "Disallow: /", "named": true, "reason": "longest matching rule is 'Disallow: /'"},
    "Googlebot": {"allowed": true, "group": "*, Googlebot", "rule": null, "named": false, "reason": "no rule matches"}
  },
  "sitemaps": {"source": "robots.txt", "count": 8, "complete": false, "incompleteReason": "maxUrls (1,000) reached", "returnedUrls": 1000},
  "urls": [{"loc": "https://www.nytimes.com/athletic/...", "lastmod": "2026-10-24T16:20:30Z"}, "..."],
  "llmsTxt": {"present": false, "httpStatus": 404, "note": "Not published (HTTP 404)."}
}

For developers.cloudflare.com the same record shows every crawler allowed and llmsTxt.present: true with the file's title, sections and links.

Pricing in plain words

You pay $0.002 per site analyzed plus $0.001 per 1,000 URLs returned. An AI-access and llms.txt audit of 100 sites with includeUrls: false costs $0.20; a site with 25,000 sitemap URLs costs $0.027. Sites that don't exist or never answer cost nothing, and maxUrls caps the cost per site. Page metadata (extractMetadata) adds $0.001 per site from 2026-11-16 and is free until then. Full table under Pricing.

Page metadata, logo and social links (extractMetadata)

Turn on extractMetadata and each site record also gets a metadata object read from the site's home page and up to metadataPages (default 2) contact, imprint or about pages it links to:

{"sites": ["apify.com", "stripe.com"], "includeUrls": false, "extractMetadata": true}

Real output for apify.com from run BUldeKL6BQDpIQEhX (2026-10-29), shortened:

"metadata": {
  "status": "ok", "url": "https://apify.com/", "httpStatus": 200,
  "title": "Apify: Marketplace of ready-to-run tools for AI", "language": "en", "themeColor": "#020202",
  "description": "Thousands of tools to automate your business. ...", "canonical": "https://apify.com",
  "metaRobots": "index,follow", "generator": null,
  "openGraph": {"title": "Apify: Marketplace of ready-to-run tools for AI", "site_name": "Apify", "type": "website",
                "image": "https://apify.com/og-image?title=Thousands+of+tools%0Afor+your+AI", "image:width": "1200"},
  "twitter": {"card": "summary_large_image", "creator": "@apify"},
  "favicon": "https://apify.com/favicon.ico?...", "appleTouchIcon": "https://apify.com/apple-icon.png?...",
  "icons": [{"url": "https://apify.com/icon0.svg?...", "rel": "icon", "sizes": "any", "type": "image/svg+xml"}, "..."],
  "logo": "https://apify.com/img/apify-logo/apify-symbol-200x200.svg", "logoSource": "json-ld",
  "logoCandidates": [{"url": "https://apify.com/img/apify-logo/apify-symbol-200x200.svg", "source": "json-ld"},
                     {"url": "https://apify.com/img/apify-logo/wordmark-white.svg", "source": "img"},
                     {"url": "https://apify.com/apple-icon.png?...", "source": "apple-touch-icon"}],
  "manifest": null,
  "organization": {"type": "Organization", "name": "Apify", "legalName": "Apify Technologies s.r.o.", "...": "..."},
  "socialProfiles": [{"network": "github", "url": "https://github.com/apify", "source": "json-ld"},
                     {"network": "linkedin", "url": "https://linkedin.com/company/apify", "source": "json-ld"},
                     {"network": "x", "url": "https://x.com/apify", "source": "json-ld"}, "..."],
  "emails": ["hello@apify.com", "support@apify.com"], "phones": [],
  "pages": [{"url": "https://apify.com/contact", "finalUrl": null, "httpStatus": 200},
            {"url": "https://apify.com/about", "finalUrl": null, "httpStatus": 200}],
  "warnings": []
}

• Logo: a JSON-LD Organization logo first, then og:logo, a header image marked "logo", the largest web app manifest icon (192 px or more), the apple-touch-icon, then a large icon. logoCandidates lists every one found with its source, so you can pick another.
• Favicon: the declared <link rel="icon">; if none is declared, /favicon.ico only when it really exists.
• Social profiles: JSON-LD sameAs plus links in the page header, footer and navigation to Facebook, Instagram, X/Twitter, LinkedIn, YouTube, TikTok, Pinterest, GitHub, Threads, Bluesky, Discord, Reddit, Medium, Telegram and others. Share buttons, posts, repositories and the platform's own pages are left out.
• Emails and phones: only what the fetched pages publish: mailto: and tel: links, emails in the visible text (including Cloudflare-obfuscated ones) and JSON-LD contact fields. Nothing is guessed, and script code is ignored.
• status: ok; blocked (robots.txt disallows the home page for our crawler, so it is not fetched); unavailable (HTTP error such as 403 from bot protection, not HTML, or no answer; note says which); skipped (the run's maximum total charge was reached). Only ok is charged.
• No tech-stack fingerprinting; the page's own generator meta tag (e.g. WordPress) is returned when present. Pages are read as served (no JavaScript), so sites that render everything in the browser return less.

Why this one

• It does not give up. Sitemaps in the wild are messy. This Actor reads gzip files whatever their Content-Type says, double-gzipped files, BOMs, missing or wrong XML namespaces, unescaped & in URLs, files cut off mid-way, plain-text sitemaps, RSS and Atom feeds, redirects across hosts, and HTML "not found" pages served with status 200. Transient errors (429, 5xx, timeouts) are retried with backoff and Retry-After.
• Partial beats failed. One broken sitemap gets its own error entry with the reason and a fix; the rest of the site is still returned. Time and size limits return what was read so far, marked complete: false with the reason.
• Huge sites are fine. Sitemaps are parsed as a stream (no 50 MB file held in memory), indexes are expanded recursively without loops, and URLs are deduplicated across all sitemaps.
• robots.txt done properly. RFC 9309 matching (exact product-token groups, longest match wins, Allow wins ties, * and $ wildcards), lenient parsing of real-world mistakes (each reported as a warning), and the RFC rules for missing (4xx = allow all) and broken (5xx = disallow all) robots.txt.
• Agent-shaped output. Every verdict says which group applied and which rule decided, so an agent can explain it.
• Polite and honest. Only files that sites publish for machines are fetched: /robots.txt, sitemaps, /llms.txt, /llms-full.txt. No page is crawled; with extractMetadata on, only the home page and up to 5 contact/about pages it links to are read, and only where robots.txt allows it. Requests identify as TinplateSiteDiscovery/1.0 (+https://tinplatelabs.github.io), are paced per host (Crawl-delay honoured up to 3 s), and sitemaps and llms files are skipped where robots.txt disallows them (sitemaps that robots.txt itself declares count as an invitation).

Input

{
  "sites": ["python.org", "developers.cloudflare.com"],
  "userAgents": ["GPTBot", "ClaudeBot", "Google-Extended", "*"],
  "maxUrls": 5000,
  "urlPrefixes": ["/workers/"]
}

Check mode:

{"mode": "check", "urlsToCheck": ["https://example.com/blog/post", "https://example.com/admin/"], "userAgents": ["GPTBot", "Googlebot"]}

Output

One dataset item per site (shortened):

{
  "input": "example.com",
  "site": "https://example.com",
  "status": "ok",
  "robotsTxt": {
    "url": "https://example.com/robots.txt", "httpStatus": 200, "status": "ok", "sizeBytes": 412,
    "groups": [
      {"userAgents": ["GPTBot", "CCBot"], "allow": [], "disallow": ["/"], "crawlDelay": null},
      {"userAgents": ["*"], "allow": ["/admin/public/"], "disallow": ["/admin/"], "crawlDelay": 2.0}
    ],
    "sitemaps": ["https://example.com/sitemap_index.xml"],
    "otherDirectives": {}, "warnings": []
  },
  "crawlerAccess": {
    "GPTBot": {"allowed": false, "group": "GPTBot, CCBot", "rule": "Disallow: /", "crawlDelay": null, "named": true, "reason": "longest matching rule is 'Disallow: /'"},
    "ClaudeBot": {"allowed": true, "group": "*", "rule": null, "crawlDelay": 2.0, "named": false, "reason": "no rule matches"}
  },
  "sitemaps": {
    "source": "robots.txt", "count": 3, "failed": 1, "complete": true, "incompleteReason": null,
    "urlsScanned": 1207, "uniqueUrls": 1180, "duplicateUrls": 27, "invalidEntries": 0,
    "matchedUrls": 1180, "returnedUrls": 1180,
    "items": [
      {"url": "https://example.com/sitemap_index.xml", "type": "sitemapindex", "httpStatus": 200, "gzip": false, "urls": 0, "sitemaps": 2, "depth": 0, "error": null},
      {"url": "https://example.com/post-sitemap.xml.gz", "type": "urlset", "httpStatus": 200, "gzip": true, "urls": 1207, "sitemaps": 0, "depth": 1, "error": null},
      {"url": "https://example.com/old-sitemap.xml", "type": null, "httpStatus": 404, "urls": 0, "sitemaps": 0, "depth": 1,
       "error": {"code": "HTTP_ERROR", "message": "HTTP 404. The sitemap URL listed for this site does not work; the site owner needs to fix or remove it."}}
    ]
  },
  "urls": [
    {"loc": "https://example.com/blog/hello", "lastmod": "2026-09-30", "changefreq": "weekly", "priority": 0.8}
  ],
  "llmsTxt": {
    "url": "https://example.com/llms.txt", "present": true, "httpStatus": 200, "sizeBytes": 1834,
    "title": "Example", "summary": "Docs for the Example API.", "details": null, "validFormat": true, "linkCount": 12,
    "sections": [{"title": "Docs", "optional": false, "notes": null,
                  "links": [{"title": "Quick start", "url": "https://example.com/start.md", "description": "install and first call"}]}],
    "content": "# Example\n\n> Docs for the Example API.\n..."
  },
  "llmsFullTxt": {"url": "https://example.com/llms-full.txt", "present": false, "httpStatus": 404, "note": "Not published (HTTP 404)."},
  "warnings": []
}

Check mode returns one item per URL:

{"input": "https://example.com/admin/", "url": "https://example.com/admin/", "status": "ok",
 "robotsTxtUrl": "https://example.com/robots.txt", "robotsTxtStatus": "ok",
 "results": {"GPTBot": {"allowed": false, "group": "GPTBot, CCBot", "rule": "Disallow: /", "named": true, "crawlDelay": null, "reason": "longest matching rule is 'Disallow: /'"}}}

A site that cannot be analyzed gives an error item, and the run carries on:

{"input": "no-such-domain.example", "status": "error",
 "error": {"code": "SITE_NOT_FOUND", "message": "The host name 'no-such-domain.example' does not resolve in DNS (...). Check the spelling of the domain; the site may not exist or may have expired."}}

• robotsTxt.status: ok, not_found (4xx: everything allowed), unreachable (5xx or no answer: everything disallowed, per RFC 9309), html (an HTML page was served: no rules).
• sitemaps.source: robots.txt (declared), common-locations (found at /sitemap.xml, /sitemap_index.xml or /wp-sitemap.xml) or none.
• Site error codes: INVALID_SITE, INVALID_URL, SITE_NOT_FOUND, SITE_UNREACHABLE, PRIVATE_ADDRESS, TIMEOUT, SPENDING_LIMIT_REACHED, INTERNAL_ERROR. Sitemap error codes: HTTP_ERROR, NOT_A_SITEMAP, PARSE_ERROR, TOO_LARGE, FETCH_FAILED, BLOCKED_BY_ROBOTS. Every message says what to do next.
• If a site's URL list is too big for one dataset item (~8 MB), it is saved in the run's key-value store and the item gives urlsKeyValueStoreKey and urlsUrl.

Pricing (pay per event)

Examples: an AI-access and llms.txt audit of 100 sites with includeUrls: false costs $0.20. A site with 25,000 URLs in its sitemaps costs $0.002 + $0.025 = $0.027. A brand kit for 1,000 company domains (includeUrls: false, extractMetadata: true) costs $3.00 from 2026-11-16 ($2.00 before). Sites that do not exist or never answer are not charged. When the run's maximum total charge is reached, URL lists are cut to what it covers and the remaining sites are marked SPENDING_LIMIT_REACHED.

Limits

• 1,000 sites (or 10,000 URLs to check) per run; up to 200,000 returned URLs per site; up to 1,000,000 URLs counted per site with includeUrls: false.
• Up to 2,000 sitemap files per site, nested up to 6 levels; 100 MB per uncompressed sitemap; 120 s per sitemap download.
• Only public addresses are fetched. Redirects: up to 5 per request.
• User-agent groups are matched by exact product token (as RFC 9309 says); Google's fallback from tokens such as Googlebot-Image to Googlebot is not applied.
• Image, video and news sitemap extensions are not returned (their page URLs are).
• llms.txt content is returned up to 100,000 characters; llms-full.txt is summarized (size, title, headings), not returned.
• Page metadata: pages up to 3 MB, HTML as served (no JavaScript rendering), 75 s per site for the whole metadata step; up to 50 emails, 20 phone numbers and 20 icon links per site.

Use from an AI agent (MCP)

Add the Apify MCP server to your agent and enable this Actor. Typical calls:

{"sites": ["example.com"], "includeUrls": false}

{"sites": ["example.com"], "urlPrefixes": ["/docs/"]}

Open-source licenses

All dependencies are permissively licensed. CI fails on any copyleft dependency.

The robots.txt parser and matcher are our own (RFC 9309). Test-only: pytest (MIT), Protego (BSD-3-Clause) as a reference to compare robots.txt verdicts against.

Made by Tinplate Labs. This Actor is maintained by an AI; there is no support desk, so every error message is written to tell you exactly what to change.