XML Sitemap Scraper & URL Extractor API - SEO Crawler

XML Sitemap Scraper & URL Extractor API

Parse XML website sitemaps and extract URLs for SEO crawls, audits, content migrations, RAG indexing, competitive research, and website monitoring. The actor can discover sitemap files from robots.txt, follow nested sitemap indexes, and export publisher-supplied URL metadata in bulk.

What it does

Give it a full website URL or sitemap.xml link, and it returns listed URLs up to your maxResults cap, with available sitemap metadata such as lastmod, changefreq, and priority. Use it as a sitemap parser, URL extraction API, or lightweight SEO crawler seed-list generator.

Key features:

• Auto-discovery -- Checks robots.txt and common sitemap paths
• Nested sitemap support -- Follows sitemap index files up to four levels including the root
• Gzip support -- Parses gzip-compressed sitemap responses
• Sitemap metadata -- Extracts lastmod, changefreq, priority, and optional image declarations
• Date filtering -- Filters parseable publisher-supplied lastmod values
• Bulk processing -- Processes multiple websites with one total output cap
• No external API key -- Uses public website and sitemap responses

Input

Field	Description	Default
urls	Full website or sitemap.xml URLs to parse	https://www.apify.com/sitemap.xml
autoDiscover	Check robots.txt and common paths for website inputs	true
maxResults	Total output cap across all inputs	25
modifiedSince	Filter parseable publisher-supplied lastmod values	(all)
includeImages	Include image declarations from sitemap extensions	false

Output

Each result contains:

{
  "url": "https://example.com/page",
  "sitemapSource": "https://example.com/sitemap.xml",
  "lastModified": "2026-04-01",
  "changeFreq": "weekly",
  "priority": "0.8",
  "images": []
}

What the output means

A sitemap is a website-publisher declaration, not an independent crawl or verification. Results show that a URL and metadata appeared in the fetched sitemap response at run time. They do not prove that a page or image is reachable, indexed by a search engine, canonical, current, safe, owned by a particular party, or unchanged since lastmod.

lastmod, changefreq, and priority are optional publisher-supplied hints. They can be absent, stale, malformed, or ignored by search engines. When modifiedSince is set, entries with a parseable date before the cutoff are excluded; entries with missing or unparseable dates remain because their modification date is unknown. Auto-discovery checks robots.txt and a small set of conventional paths, so an empty result does not prove that no sitemap exists.

Use cases

• SEO Audit / Crawling -- Get a complete list of sitemap URLs with metadata before a crawl
• RAG Indexing -- Build URL seed lists for documentation, help centers, and content ingestion
• Content Migration -- Map all URLs before migrating to a new platform
• Competitive Analysis -- See what pages competitors have and how often they update
• Website Monitoring -- Track new pages added since a specific date
• Link Building -- Find all pages on a target site to identify link opportunities
• Content Inventory -- Catalog all pages with their update frequency and priority

Pricing

The live pay-per-result rate is $0.0005 per extracted URL: about $0.0125 for the 25-result default or $0.50 per 1,000 results. Set maxResults to control the total output and charge. Account-specific platform charges or credits may apply.

How to use

1. Open XML Sitemap Scraper & URL Extractor API - SEO Crawler in Apify Console and click Start (or Try for free).
2. Fill in the input form - start with Website or Sitemap URLs, Auto-Discover Sitemaps, Max Results. Every field is optional unless marked required, and the form is prefilled with a working example.
3. Set Maximum results to cap your cost, then run the Actor.
4. Download the results as JSON, CSV, Excel, XML or HTML, or pull them with the Apify API.

Example input

{
  "urls": [
    "https://www.apify.com/sitemap.xml"
  ],
  "autoDiscover": true,
  "maxResults": 100,
  "includeImages": false
}

Output example

One record from a real run (long values shortened, contact details partly masked):

{
  "url": "https://apify.com/",
  "sitemapSource": "https://apify.com/sitemap/pages.xml"
}

FAQ

How much does XML Sitemap Scraper & URL Extractor API - SEO Crawler cost?

$0.0005 per url extracted. For example, 1,000 results cost about $0.50. Higher Apify plans get a Store discount.

Do I need an API key or login?

No. Just configure the input and run it.

Can I run it on a schedule or from code?

Yes. Use Apify Schedules for recurring runs, or call it from the Apify API, the JavaScript or Python client, Make, Zapier, n8n, or any MCP-compatible AI agent.

Why did my run return no results?

Your filters are probably too narrow. Remove a filter or broaden the search, then run again. You only pay the result price for records that are actually saved.

Is the data accurate and up to date?

Each run fetches live data from the source described above, so results are as current as that source. Verify critical records with the original source before acting on them.