Sitemap Extractor & Site Map: robots.txt, RSS & llms.txt

Site Discovery for AI Agents: sitemaps, robots.txt, RSS & llms.txt

Give it a list of domains. It maps each site without crawling a single page: what robots.txt allows for your user agent, every sitemap (including sitemap indexes and gzipped .xml.gz files), RSS/Atom/JSON feeds, and whether the site publishes llms.txt / llms-full.txt for AI agents. You can also get a clean, filtered URL list with lastmod, changefreq and priority.

It is built for teams that feed websites to LLMs, RAG pipelines, AI agents and SEO tools. You find out what to fetch, what has changed since your last run, and what you are allowed to fetch before you spend money on a crawler.

Use cases

• Crawler and RAG seeding: get the complete URL list from sitemaps before you crawl or embed.
• SEO audits: find missing or broken sitemaps, sitemap size and lastmod freshness.
• AI-bot policy research: see which sites block GPTBot, ClaudeBot or CCBot in robots.txt, and who publishes llms.txt.
• Content monitoring: pull only URLs with lastmod after a date to see what a site published this week.
• Feed discovery: find RSS/Atom/JSON feeds for news and blog monitoring.

What you get per domain

• robots.txt summary for the user agent you choose (for example GPTBot, Googlebot or your own bot). It shows the matched group, the allow and disallow rules, crawl-delay, whether the whole site is blocked for you, and which well-known AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot…) are blocked. Rules are matched per RFC 9309: longest match wins, with * and $ wildcards.
• Sitemaps listed in robots.txt. When robots.txt lists none, the common paths are tried. Sitemap indexes are followed, newest first, and gzip is handled.
• Feeds from <link rel="alternate"> on the homepage, or from common paths such as /feed, /rss.xml and /atom.xml. Each feed shows its format, title, item count and latest item date.
• llms.txt and llms-full.txt: whether each is present, its size, and an excerpt.
• URL counts: URLs scanned, URLs that matched your filters, URLs filtered out, URLs disallowed by robots.txt, and URLs output.

There is also an optional per-URL record: url, lastmod, changefreq, priority, sourceSitemap and allowedByRobots.

Filters

• includePatterns / excludePatterns match the URL path. Use a glob (/blog/**, /docs/*/intro) or a regex with the re: prefix (re:^/p/\d+).
• lastmodSince keeps only URLs modified on or after a date. Child sitemaps older than that date are skipped. URLs without a lastmod are dropped when this is set.
• maxUrls caps the URL records per domain. maxSitemaps caps the sitemap files fetched per domain.

Output sample (real test run, trimmed)

{
 "recordType": "domain-summary", "domain": "vercel.com", "homepageStatus": 200,
 "robots": { "status": 200, "present": true, "userAgent": "GPTBot", "matchedGroup": "*",
   "disallow": ["/api/", "/oauth", "/confirm", "..."], "allow": ["/api/og/*", "..."],
   "crawlDelay": null, "blocksEverything": false, "aiBotsBlocked": [] },
 "sitemaps": [ { "url": "https://vercel.com/sitemap.xml", "status": 200, "type": "urlset", "entries": 9269, "error": null } ],
 "feeds": [ { "url": "https://vercel.com/atom", "foundVia": "common-path", "format": "atom", "title": "Vercel News", "itemCount": 2102, "latestItem": "2026-10-29T18:44:00.000Z" } ],
 "llmsTxt": { "present": true, "bytes": 5397, "excerpt": "# Vercel\n\n> Vercel is a cloud platform for building, deploying, and scaling web applications and AI workloads. ..." },
 "llmsFullTxt": { "present": false },
 "urlCounts": { "matchedFilters": 30, "disallowedByRobots": 0, "output": 30, "truncatedAtMaxUrls": true },
 "error": null
}
{ "recordType": "url", "domain": "vercel.com", "url": "https://vercel.com/connect/ocean-io",
  "lastmod": "2026-10-25T14:10:15.167Z", "changefreq": null, "priority": null,
  "sourceSitemap": "https://vercel.com/sitemap.xml", "allowedByRobots": true }

The Domain summaries and URLs dataset views split the two record types.

Pricing (pay per event)

Event	Price
domain-summary: one reachable domain analyzed	$0.002
url-discovered: one per-URL record written	$0.0003

Failures are never charged. A domain where we could not get robots.txt, the homepage or any sitemap is still listed with an error, but it is free. URLs dropped by your filters or by robots.txt are free too. Cost example: 100 domains with 500 URLs each = 100 × $0.002 + 50,000 × $0.0003 = $15.20. With summaries only (outputUrls: false), the same run costs $0.20.

Limitations

• Only one origin is analyzed per input. wordpress.org/news analyzes wordpress.org, so enter subdomains such as blog.example.com separately.
• Sites that block datacenter IPs may return 403 for sitemaps. Those sitemaps are listed with their error.
• When lastmodSince is set and a site has no lastmod values (common), every URL is filtered out. Check urlCounts.filteredOut.

Use it from AI agents (MCP), Zapier, Make, n8n or the API

AI agents (Claude, Cursor, VS Code, ChatGPT and other MCP clients). Add Apify's hosted MCP server with just this Actor as a tool:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=steadyline/site-discovery"
    }
  }
}

Sign in with Apify (OAuth) on first connect, or add an Authorization: Bearer <APIFY_TOKEN> header. Then ask your agent, for example:

Use steadyline/site-discovery to list all blog URLs changed in the last 30 days on vercel.com and tell me whether the site blocks GPTBot.

The agent sees this Actor's input schema and output fields, so it can call it without extra prompting. Agents that use the general Apify MCP server can also find it with search-actors.

This Actor uses pay-per-event pricing with limited permissions, so it can also be run through Apify's agentic payments (no Apify account needed for the agent). See the Apify docs.

Zapier, Make and n8n. Use the official Apify app: Zapier "Run Actor" action, Make "Run an Actor" module, or the n8n Apify node. Choose steadyline/site-discovery, paste the input below, then use "Get dataset items" (or the run's dataset) to send results to Google Sheets, Slack, a CRM or a database.

API (any language). Run it and get the results in one call:

curl -X POST "https://api.apify.com/v2/acts/steadyline~site-discovery/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["apify.com", "vercel.com", "blog.cloudflare.com"], "maxUrls": 100, "userAgent": "GPTBot"}'

Or with the Python client (pip install apify-client):

from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("steadyline/site-discovery").call(run_input={"domains": ["apify.com", "vercel.com", "blog.cloudflare.com"], "maxUrls": 100, "userAgent": "GPTBot"})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Works well with other Steadyline Data tools

• Website → RAG knowledge base: list URLs here, convert PDFs with PDF to Markdown and HTML pages with HTTP Request Runner (outputMarkdown: true). About $0.61 for 500 URLs.
• Map, profile, contact: this Actor → Tech Stack Detector → Website Contact Extractor.
• Related: PDF & Document to Markdown, Bulk HTTP Request Runner, Tech Stack Detector.

FAQ

Does it crawl the site? No. It reads robots.txt, sitemaps (including indexes and .gz files), feeds and llms.txt only, so it is fast and light on the target site.

Can I check whether a site blocks AI crawlers? Yes. Set userAgent (e.g. GPTBot, ClaudeBot, CCBot) and the summary shows the robots.txt rules that apply, and each URL gets allowedByRobots.

What does it cost? $0.002 per domain summary plus $0.0003 per URL you export. Set outputUrls: false for summaries only.

Can I run it on a schedule? Yes. Create an Apify Schedule (or a saved Task with a schedule) and connect a webhook or integration to receive results.

Need a feature or found a bug? Open an issue on the Issues tab of this page or email steadylinedata@proton.me. Built and maintained by Steadyline Data.

Acceptable use

This Actor reads only public, machine-readable files that sites publish for robots (robots.txt, sitemaps, feeds, llms.txt), plus the homepage. It identifies itself with a descriptive user agent. Respecting robots.txt is on by default. Use the results in line with each site's terms and robots.txt, and don't use them to overload or attack sites. You are responsible for what you do with the discovered URLs.

Made by Steadyline Data.