# Sitemap Extractor & Site Map: robots.txt, RSS & llms.txt (`steadyline/site-discovery`) Actor

Map any website without crawling: robots.txt rules for your user agent (incl. AI-bot blocks like GPTBot), every sitemap and sitemap index (.gz too), RSS/Atom/JSON feeds, llms.txt and llms-full.txt, plus a filtered URL list with lastmod. By Steadyline Data.

- **URL**: https://apify.com/steadyline/site-discovery.md
- **Developed by:** [Steadyline Data](https://apify.com/steadyline) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 domain summaries

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Site Discovery for AI Agents: sitemaps, robots.txt, RSS & llms.txt

Give it a list of domains. It maps each site **without crawling a single page**: what robots.txt allows for *your* user agent, every sitemap (including sitemap indexes and gzipped `.xml.gz` files), RSS/Atom/JSON feeds, and whether the site publishes `llms.txt` / `llms-full.txt` for AI agents. You can also get a clean, filtered URL list with `lastmod`, `changefreq` and `priority`.

It is built for teams that feed websites to LLMs, RAG pipelines, AI agents and SEO tools. You find out what to fetch, what has changed since your last run, and what you are allowed to fetch before you spend money on a crawler.

### Use cases

- **Crawler and RAG seeding:** get the complete URL list from sitemaps before you crawl or embed.
- **SEO audits:** find missing or broken sitemaps, sitemap size and lastmod freshness.
- **AI-bot policy research:** see which sites block GPTBot, ClaudeBot or CCBot in robots.txt, and who publishes llms.txt.
- **Content monitoring:** pull only URLs with lastmod after a date to see what a site published this week.
- **Feed discovery:** find RSS/Atom/JSON feeds for news and blog monitoring.

### What you get per domain

- **robots.txt summary** for the user agent you choose (for example `GPTBot`, `Googlebot` or your own bot). It shows the matched group, the allow and disallow rules, crawl-delay, whether the whole site is blocked for you, and which well-known AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot…) are blocked. Rules are matched per RFC 9309: longest match wins, with `*` and `$` wildcards.
- **Sitemaps** listed in robots.txt. When robots.txt lists none, the common paths are tried. Sitemap indexes are followed, newest first, and gzip is handled.
- **Feeds** from `<link rel="alternate">` on the homepage, or from common paths such as `/feed`, `/rss.xml` and `/atom.xml`. Each feed shows its format, title, item count and latest item date.
- **llms.txt and llms-full.txt**: whether each is present, its size, and an excerpt.
- **URL counts**: URLs scanned, URLs that matched your filters, URLs filtered out, URLs disallowed by robots.txt, and URLs output.

There is also an optional **per-URL record**: `url`, `lastmod`, `changefreq`, `priority`, `sourceSitemap` and `allowedByRobots`.

### Filters

- `includePatterns` / `excludePatterns` match the URL path. Use a glob (`/blog/**`, `/docs/*/intro`) or a regex with the `re:` prefix (`re:^/p/\d+`).
- `lastmodSince` keeps only URLs modified on or after a date. Child sitemaps older than that date are skipped. URLs without a lastmod are dropped when this is set.
- `maxUrls` caps the URL records per domain. `maxSitemaps` caps the sitemap files fetched per domain.

### Output sample (real test run, trimmed)

```json
{
 "recordType": "domain-summary", "domain": "vercel.com", "homepageStatus": 200,
 "robots": { "status": 200, "present": true, "userAgent": "GPTBot", "matchedGroup": "*",
   "disallow": ["/api/", "/oauth", "/confirm", "..."], "allow": ["/api/og/*", "..."],
   "crawlDelay": null, "blocksEverything": false, "aiBotsBlocked": [] },
 "sitemaps": [ { "url": "https://vercel.com/sitemap.xml", "status": 200, "type": "urlset", "entries": 9269, "error": null } ],
 "feeds": [ { "url": "https://vercel.com/atom", "foundVia": "common-path", "format": "atom", "title": "Vercel News", "itemCount": 2102, "latestItem": "2026-10-29T18:44:00.000Z" } ],
 "llmsTxt": { "present": true, "bytes": 5397, "excerpt": "# Vercel\n\n> Vercel is a cloud platform for building, deploying, and scaling web applications and AI workloads. ..." },
 "llmsFullTxt": { "present": false },
 "urlCounts": { "matchedFilters": 30, "disallowedByRobots": 0, "output": 30, "truncatedAtMaxUrls": true },
 "error": null
}
{ "recordType": "url", "domain": "vercel.com", "url": "https://vercel.com/connect/ocean-io",
  "lastmod": "2026-10-25T14:10:15.167Z", "changefreq": null, "priority": null,
  "sourceSitemap": "https://vercel.com/sitemap.xml", "allowedByRobots": true }
```

The **Domain summaries** and **URLs** dataset views split the two record types.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `domain-summary`: one reachable domain analyzed | $0.002 |
| `url-discovered`: one per-URL record written | $0.0003 |

**Failures are never charged.** A domain where we could not get robots.txt, the homepage or any sitemap is still listed with an `error`, but it is free. URLs dropped by your filters or by robots.txt are free too.
**Cost example:** 100 domains with 500 URLs each = 100 × $0.002 + 50,000 × $0.0003 = **$15.20**. With summaries only (`outputUrls: false`), the same run costs **$0.20**.

### Limitations

- Only one origin is analyzed per input. `wordpress.org/news` analyzes `wordpress.org`, so enter subdomains such as `blog.example.com` separately.
- Sites that block datacenter IPs may return 403 for sitemaps. Those sitemaps are listed with their error.
- When `lastmodSince` is set and a site has no lastmod values (common), every URL is filtered out. Check `urlCounts.filteredOut`.

### Use it from AI agents (MCP), Zapier, Make, n8n or the API

**AI agents (Claude, Cursor, VS Code, ChatGPT and other MCP clients).** Add Apify's hosted MCP server with just this Actor as a tool:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=steadyline/site-discovery"
    }
  }
}
```

Sign in with Apify (OAuth) on first connect, or add an `Authorization: Bearer <APIFY_TOKEN>` header. Then ask your agent, for example:

> Use steadyline/site-discovery to list all blog URLs changed in the last 30 days on vercel.com and tell me whether the site blocks GPTBot.

The agent sees this Actor's input schema and output fields, so it can call it without extra prompting. Agents that use the general Apify MCP server can also find it with `search-actors`.

This Actor uses pay-per-event pricing with limited permissions, so it can also be run through Apify's agentic payments (no Apify account needed for the agent). See the [Apify docs](https://docs.apify.com/platform/integrations/mcp#agentic-payments).

**Zapier, Make and n8n.** Use the official Apify app: Zapier "Run Actor" action, Make "Run an Actor" module, or the n8n Apify node. Choose `steadyline/site-discovery`, paste the input below, then use "Get dataset items" (or the run's dataset) to send results to Google Sheets, Slack, a CRM or a database.

**API (any language).** Run it and get the results in one call:

```bash
curl -X POST "https://api.apify.com/v2/acts/steadyline~site-discovery/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["apify.com", "vercel.com", "blog.cloudflare.com"], "maxUrls": 100, "userAgent": "GPTBot"}'
```

Or with the Python client (`pip install apify-client`):

```python
from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("steadyline/site-discovery").call(run_input={"domains": ["apify.com", "vercel.com", "blog.cloudflare.com"], "maxUrls": 100, "userAgent": "GPTBot"})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

### Works well with other Steadyline Data tools

- **Website → RAG knowledge base:** list URLs here, convert PDFs with [PDF to Markdown](https://apify.com/steadyline/pdf-document-to-markdown) and HTML pages with [HTTP Request Runner](https://apify.com/steadyline/http-request-runner) (`outputMarkdown: true`). About $0.61 for 500 URLs.
- **Map, profile, contact:** this Actor → [Tech Stack Detector](https://apify.com/steadyline/tech-stack-detector) → [Website Contact Extractor](https://apify.com/steadyline/website-contact-extractor).
- Related: [PDF & Document to Markdown](https://apify.com/steadyline/pdf-document-to-markdown), [Bulk HTTP Request Runner](https://apify.com/steadyline/http-request-runner), [Tech Stack Detector](https://apify.com/steadyline/tech-stack-detector).

### FAQ

**Does it crawl the site?**
No. It reads robots.txt, sitemaps (including indexes and .gz files), feeds and llms.txt only, so it is fast and light on the target site.

**Can I check whether a site blocks AI crawlers?**
Yes. Set `userAgent` (e.g. `GPTBot`, `ClaudeBot`, `CCBot`) and the summary shows the robots.txt rules that apply, and each URL gets `allowedByRobots`.

**What does it cost?**
$0.002 per domain summary plus $0.0003 per URL you export. Set `outputUrls: false` for summaries only.

**Can I run it on a schedule?** Yes. Create an Apify Schedule (or a saved Task with a schedule) and connect a webhook or integration to receive results.

**Need a feature or found a bug?** Open an issue on the Issues tab of this page or email steadylinedata@proton.me. Built and maintained by **Steadyline Data**.

### Acceptable use

This Actor reads only public, machine-readable files that sites publish for robots (robots.txt, sitemaps, feeds, llms.txt), plus the homepage. It identifies itself with a descriptive user agent. Respecting robots.txt is on by default. Use the results in line with each site's terms and robots.txt, and don't use them to overload or attack sites. You are responsible for what you do with the discovered URLs.

Made by **Steadyline Data**.

# Actor input Schema

## `domains` (type: `array`):

Domains (example.com) or URLs. Each origin is analyzed once.

## `outputUrls` (type: `boolean`):

Write one dataset row per discovered URL (charged per URL). Off = domain summaries only.

## `maxUrls` (type: `integer`):

Stop collecting URLs for a domain after this many (after filters).

## `includePatterns` (type: `array`):

Only keep URLs whose path matches one of these. Glob on path (/blog/\*\*, /docs/\*/intro) or regex with re: prefix (re:^/p/\d+).

## `excludePatterns` (type: `array`):

Drop URLs whose path matches (same syntax).

## `lastmodSince` (type: `string`):

Only keep URLs with a sitemap lastmod on/after this date. URLs without lastmod are dropped when set.

## `followSitemapIndexes` (type: `boolean`):

Recurse into sitemap index files (including gzipped .xml.gz).

## `maxSitemaps` (type: `integer`):

Safety cap on sitemap files fetched per domain.

## `discoverFeeds` (type: `boolean`):

From <link rel=alternate> on the homepage, else common paths (/feed, /rss.xml, /atom.xml...).

## `fetchLlmsTxt` (type: `boolean`):

Check /llms.txt and /llms-full.txt and include an excerpt.

## `llmsExcerptChars` (type: `integer`):

Characters of llms.txt content to include.

## `userAgent` (type: `string`):

User-agent token whose robots.txt rules are evaluated (e.g. GPTBot, Googlebot, MyAgent).

## `respectRobots` (type: `boolean`):

Drop URLs disallowed for the user agent above (counted in urlCounts.disallowedByRobots). Off = output them with allowedByRobots=false.

## `maxConcurrency` (type: `integer`):

Max 20.

## `timeoutSecs` (type: `integer`):

Per request.

## Actor input object example

```json
{
  "domains": [
    "apify.com",
    "vercel.com",
    "blog.cloudflare.com"
  ],
  "outputUrls": true,
  "maxUrls": 100,
  "includePatterns": [],
  "excludePatterns": [],
  "followSitemapIndexes": true,
  "maxSitemaps": 50,
  "discoverFeeds": true,
  "fetchLlmsTxt": true,
  "llmsExcerptChars": 2000,
  "userAgent": "GPTBot",
  "respectRobots": true,
  "maxConcurrency": 5,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

One row per domain (rows with recordType=url are blank here).

## `urls` (type: `string`):

Per-URL records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com",
        "vercel.com",
        "blog.cloudflare.com"
    ],
    "maxUrls": 100,
    "userAgent": "GPTBot"
};

// Run the Actor and wait for it to finish
const run = await client.actor("steadyline/site-discovery").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "apify.com",
        "vercel.com",
        "blog.cloudflare.com",
    ],
    "maxUrls": 100,
    "userAgent": "GPTBot",
}

# Run the Actor and wait for it to finish
run = client.actor("steadyline/site-discovery").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com",
    "vercel.com",
    "blog.cloudflare.com"
  ],
  "maxUrls": 100,
  "userAgent": "GPTBot"
}' |
apify call steadyline/site-discovery --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steadyline/site-discovery"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TJEhxMowD2D4hBrIb/builds/SZMxg41XWbmXQw4Mt/openapi.json