# Sitemap Extractor (`apify/sitemap-extractor`) Actor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

- **URL**: https://apify.com/apify/sitemap-extractor.md
- **Developed by:** [Apify](https://apify.com/apify) (Apify)
- **Categories:** Developer tools, SEO tools, Open source
- **Features:** Limited permissions
- **Stats:** 455 total users, 53 monthly users, 91.3% runs succeeded, 8 bookmarks
- **User rating**: 3.24 out of 5 stars

## Pricing

from $0.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor

This Actor is designed to bridge the gap between discovery and crawling. By traversing a website's `sitemap.xml` structure, it compiles a comprehensive list of all published pages and verifies their status before you commit resources to a full-scale scrape.

### Features

- **Recursive Sitemap Discovery:** Automatically detects and traverses nested sitemaps (sitemap indexes).
- **Efficiency:** Uses HTTP HEAD requests for URL validation, which are significantly faster and consume less bandwidth than full GET requests.
- **Proxy Support:** Integrated with Apify Proxy to prevent rate limiting or blocking during the discovery phase.
- **Detailed Output:** Provides the final URL, the corresponding HTTP status code, and the date-time of the page's last modification.

### How it Works

1. **Input:** You provide one or more "Start URLs" pointing to the domain name root, sitemaps or sitemap indexes.
2. **Extraction:** The Actor parses the XML, extracting both page URLs and links to further sitemaps.
3. **Validation:** For every page URL found, the Actor performs a status check (unless **Check page status** is disabled, see below).
4. **Deduplication:** The crawler uses unique keys to ensure that even if a URL appears in multiple sitemaps, it is only checked once.

### Listing sitemap URLs without status checks

By default (**Check page status** enabled), the Actor sends an HTTP HEAD request to every page found in the sitemaps and reports its status code. If you only need the list of URLs contained in the sitemaps and don't care about status codes, disable the **Check page status** option. In this mode the Actor:

- does **not** send any request to the individual pages. It only downloads and parses the sitemaps themselves;
- outputs each discovered URL with the `lastmod` value from the sitemap's `<lastmod>` tag, whenever the sitemap provides one. Since no pages are requested, the `Last-Modified` header fallback is not used, so URLs whose sitemap entry has no `<lastmod>` will have `lastmod: null`;
- omits the `status` field from the output.

This is dramatically faster and lighter on large sites, since it replaces one request per page with just a handful of requests for the sitemap files.

### Output

For each page URL, the Actor outputs:

| Field    |                                       Description                                       |
| :------- | :-------------------------------------------------------------------------------------: |
| `url`    |                                The page URL from the sitemap.                           |
| `status` | The HTTP status code returned by the HEAD request. Omitted when **Check page status** is disabled. |
| `lastmod` | Best-effort last-modification time (ISO 8601). See the note below.                      |

#### A note on last-modification data

The `lastmod` field is a single best-effort timestamp derived from two sources, in this order of preference:

1. The `<lastmod>` tag declared for the URL in the sitemap.
2. The `Last-Modified` HTTP header returned by the page (used only when the sitemap has no `<lastmod>`).

**We cannot guarantee that this information is available.** Both sources are optional: many sitemaps omit `<lastmod>` entirely, and a lot of servers don't send a `Last-Modified` header (this is especially common for dynamically generated pages). When neither source provides a value, `lastmod` is `null`. Even when present, the value is self-reported by the site and may not reflect the true last-modification time of the content.

When **Check page status** is disabled, no page requests are made, so only the first source (the sitemap's `<lastmod>` tag) is used.

### Usage

This Actor is ideal for:

- **Pre-crawling filter:** Generating a "clean" list of URLs for actors like *Website Content Crawler* or *Web Scraper*.
- **SEO Audits:** Quickly identifying 404 Not Found or 500 Server Error pages listed in your sitemap.
- **Site Mapping:** Getting a high-level overview of a site's architecture.

### Configuration

|          Field          |                           Description                           |
| :---------------------: | :-------------------------------------------------------------: |
|     **Start URLs**      | Just a domain name or a list of sitemap XML URLs to start from. |
| **Proxy configuration** |                   Settings for Apify Proxies.                   |

# Changelog

This Actor's version history is a separate document: https://apify.com/apify/sitemap-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

A static list of domains to scrape. <br><br>For details, see the <a href='https://apify.com/apify/sitemap-extractor#start-urls' target='_blank' rel='noopener'>Start URLs</a> section in the README.

## `checkPageStatus` (type: `boolean`):

When enabled (the default), the Actor sends a HEAD request to every page found in the sitemaps and outputs its HTTP status code. Disable this option to skip those requests entirely and simply output the list of page URLs (with <code>lastmod</code>) discovered in the sitemaps. This is much faster and lighter, but the output will not contain the <code>status</code> field, and <code>lastmod</code> falls back to the sitemap only (no <code>Last-Modified</code> header), so it will be <code>null</code> more often.

## `proxyConfiguration` (type: `object`):

Specifies proxy servers that will be used by the scraper in order to hide its origin.<br><br>For details, see <a href='https://apify.com/apify/sitemap-extractor#proxy-configuration' target='_blank' rel='noopener'>Proxy configuration</a> in README.

## `proxyRotation` (type: `string`):

This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.

## `maxCrawlingDepth` (type: `integer`):

Specifies how many sitemap levels the crawler will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers. <br><br>If set to <code>1</code>, will process only the root sitemap. If set to <code>0</code>, there is no limit.

## `maxRequestRetries` (type: `integer`):

The maximum number of times the scraper will retry to load each web page on error, in case of a page load error or an exception thrown by the <b>Page function</b>.<br><br>If set to <code>0</code>, the page will be considered failed right after the first error.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "checkPageStatus": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "proxyRotation": "RECOMMENDED",
  "maxCrawlingDepth": 0,
  "maxRequestRetries": 3
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items with discovered URLs, crawl status, and date-time of last modification.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("apify/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("apify/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call apify/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,apify/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rGeTNESChDZ65EbYh/builds/QwbRyX1NPjZC6gCae/openapi.json