Bulk URL Checker — Status, Redirects & SEO Tags

🔗 Bulk URL Checker — HTTP Status, Redirects, Response Time & SEO Tags at Scale

Check thousands — or millions — of URLs at once. This bulk URL checker takes a pasted list, an uploaded file, seed domains, or a remote text/CSV/XML sitemap of links and returns, for every URL: HTTP status code, redirect status, final destination URL, response time (ms), HTTPS check, Content-Type, response size (Content-Length), server header, plus on-page SEO tags — <title>, meta description, <h1>, canonical URL, Open Graph (og:title, og:image), meta robots, isNoindex flag and body word count.

Export every checked URL to JSON, CSV, Excel, or API — built for 1,000+ URL bulk audits with high concurrency and zero duplicates.

Built for technical SEO specialists, link builders, QA & site-migration engineers, webmasters, B2B lead-list cleaners and data engineering teams who need fast, reliable bulk URL validation and on-page metadata extraction without headless-browser overhead.

---

⚡ At a glance

Feature	Details
Input sources	Pasted URLs, bare domains (example.com), uploaded text/CSV files, or remote XML sitemaps (requestsFromUrl)
HTTP & transport metrics	statusCode (200, 301, 404, 500…), ok (true/false), finalUrl, redirected, responseTimeMs, isHttps, contentType, contentLength, server, error
On-page SEO signals	title, metaDescription, h1, canonical, ogTitle, ogImage, metaRobots, isNoindex, wordCount (toggleable via checkSeo)
Content types supported	HTML pages, PDFs, images, JSON/XML feeds and binary CDN assets (*/*)
Speed & concurrency	Pure HTTP with configurable concurrency (1–500, default 50) — checks 100 URLs in seconds and 1,000+ URLs in minutes
Scale & link discovery	Checks your exact list (unlimited when maxItems: 0), or automatically discovers page links when maxItems exceeds your seed URLs
Deduplication & storage	Strictly deduplicated by normalized URL; writes dataset items plus Key-Value Store OUTPUT and SUMMARY records
Export formats	Bulk export to JSON, CSV, Excel, XML, JSONL, Webhook & Apify REST API

---

🔎 Why use a bulk URL checker?

Checking a single URL in a browser tab takes two seconds. Checking an entire XML sitemap, a 10,000-row backlink export, a product catalog of 50,000 landing pages, or a B2B lead list of company domains requires parallel HTTP execution, redirect resolution, timeout handling, SSL tolerance, and structured output. Spreadsheet formulas (IMPORTXML) and single-threaded scripts choke on slow servers, drop failed requests, or get stuck on non-HTML files.

This bulk URL checker and SEO tag extractor solves that at cloud scale: point it at your URLs (or a remote sitemap / seed domain) and get one clean, deduplicated row per URL showing its exact HTTP status, where it redirected, how fast it responded, its MIME type and byte size, and its core on-page SEO tags. Because it runs on high-concurrency plain HTTP rather than a heavy browser, large runs finish rapidly at a fraction of a cent per thousand URLs.

Common reasons teams use this bulk URL checker:

• 🔍 Technical SEO & sitemap audits — extract title, metaDescription, h1, canonical and isNoindex across thousands of URLs to catch missing, duplicate or de-indexed pages.
• 🚨 Broken link & 404 detection — filter by ok: "false" to isolate every 404, 410, 500, 502, DNS failure and timeout across internal links or external backlinks.
• 🔀 Site migration & redirect verification — audit redirected and finalUrl before and after a domain or CMS migration to ensure old URLs resolve to the right 200 destinations.
• 🧹 B2B lead & domain list cleaning — validate thousands of company websites before sales outreach or deeper scraping, stripping out dead domains and resolving parked redirects.
• 🖼️ CDN, image & PDF asset verification — confirm non-HTML assets (image/webp, application/pdf, application/xml) return 200 OK and expected contentLength.
• ⏱️ Uptime & latency monitoring — schedule recurring runs to track statusCode and responseTimeMs across critical landing pages and API endpoints.

---

✨ What this bulk URL checker does

• ✅ Accepts any URL list size — paste URLs directly, upload a file, or link a remote text, CSV or XML sitemap (requestsFromUrl) containing thousands to millions of URLs.
• ✅ Smart seed-to-1,000+ link expansion — if you pass a few seed URLs (e.g. a homepage) and set maxItems: 50 or maxItems: 1000, the Actor checks your seed URLs and automatically follows discovered links up to your exact maxItems target. When maxItems: 0 (the default), it checks only the exact URLs you supplied.
• ✅ 100-URL built-in demo on empty runs — running with empty input ({}) checks a curated 100-URL sample of major documentation, news and reference sites in under 30 seconds.
• ✅ High-speed parallel execution — configurable maxConcurrency up to 500 parallel connections.
• ✅ Complete HTTP & redirect diagnostics — reports statusCode, ok, finalUrl, redirected, responseTimeMs, isHttps, contentType, contentLength, server and error.
• ✅ Works on HTML and non-HTML URLs alike — PDFs, images, JSON endpoints and feeds are verified with their real HTTP status and headers instead of being rejected.
• ✅ On-page SEO extraction (checkSeo: true) — parses <title>, <meta name="description">, first <h1>, <link rel="canonical">, og:title, og:image, <meta name="robots">, isNoindex and body wordCount.
• ✅ Zero silent drops & zero duplicates — unreachable URLs are logged with ok: "false" and a clear error message, and every URL is deduplicated by normalized href.
• ✅ Key-Value Store OUTPUT & SUMMARY mirror — in addition to streaming items to the default Dataset, the Actor writes OUTPUT and SUMMARY (checked, okCount, redirectedCount, errorCount) to the Key-Value Store for instant API retrieval.

---

🎯 Use cases

Persona	How they use this Actor
📈 Technical SEO Auditors	Point requestsFromUrl at a client's sitemap.xml to audit 1,000+ pages for missing titles/H1s, mismatched canonicals, noindex leaks and thin wordCount.
🚀 Site Migration & QA Teams	Upload legacy URL lists during a redesign or domain move to verify every URL returns 200 at the expected finalUrl.
🧲 B2B Lead Gen & RevOps Teams	Clean 10,000+ prospect domains from CRM exports (checkSeo: false, maxConcurrency: 200) to filter out dead websites and capture live finalUrl domains.
🔗 Link Builders & Digital PR	Verify backlink portfolios and guest-post placements in bulk to confirm pages are live (statusCode: "200") and indexable (isNoindex: "false").
🛒 E-Commerce & Catalog Managers	Validate thousands of product page URLs and CDN image links (contentType, contentLength) to catch broken product media before ad campaigns run.

---

📥 Input configuration

All fields are optional. Leave urls empty for a 100-URL demo run, or use the prefill (maxItems: 50) for a quick 50-URL test run.

Field	Type	Default	Description
urls	array	Prefill 3 seed URLs	URLs to check. Paste URLs/domains, upload a file, or link a remote text/CSV/XML sitemap via requestsFromUrl. Bare domains (example.com) are auto-prefixed with https://.
checkSeo	boolean	true	Extract on-page SEO tags (title, metaDescription, h1, canonical, ogTitle, ogImage, metaRobots, isNoindex, wordCount) on HTML responses. Turn false for pure status/redirect speed.
maxItems	integer	0 (prefill 50)	Maximum URLs to check. 0 = no limit (checks every URL provided, or 100 demo URLs on an empty run). When set higher than your seed URLs, discovered page links are checked up to maxItems.
maxConcurrency	integer	50	Parallel HTTP requests (1–500). Higher is faster; lower is gentler on a single origin server.
timeoutSecs	integer	30	Per-URL request timeout in seconds (3–120).
proxyConfiguration	object	{ "useApifyProxy": false }	Optional Apify Proxy. Off by default for lowest cost; enable for geo-specific checks or single-host rate-limit protection.

Input example — check an exact list of URLs & bare domains

{
  "urls": [
    { "url": "https://apify.com" },
    { "url": "https://docs.apify.com/platform" },
    { "url": "github.com/apify/crawlee" },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Status" }
  ],
  "checkSeo": true,
  "maxItems": 0,
  "maxConcurrency": 50,
  "timeoutSecs": 30
}

Input example — audit 1,000+ URLs from a remote XML sitemap

{
  "urls": [
    { "requestsFromUrl": "https://blog.apify.com/sitemap-posts.xml" }
  ],
  "checkSeo": true,
  "maxItems": 1000,
  "maxConcurrency": 50,
  "timeoutSecs": 30
}

Input example — ultra-fast status-only domain validation

{
  "urls": [
    { "url": "https://example.com" },
    { "url": "https://httpbin.org/status/404" }
  ],
  "checkSeo": false,
  "maxItems": 0,
  "maxConcurrency": 200,
  "timeoutSecs": 15
}

---

📤 Output & field reference

Every checked URL produces one flat record in the Dataset (and in the Key-Value Store OUTPUT array). Below is a real output item:

{
  "url": "https://blog.apify.com/shopify-competitor-analysis/",
  "statusCode": "200",
  "ok": "true",
  "finalUrl": "https://blog.apify.com/shopify-competitor-analysis/",
  "redirected": "false",
  "responseTimeMs": "372",
  "isHttps": "true",
  "contentType": "text/html",
  "contentLength": "36692",
  "server": "openresty",
  "title": "Shopify competitor analysis with Apify and Claude",
  "metaDescription": "Will your margins survive a market's import duties? What do local buyers complain about? Build an analysis report that answers these questions.",
  "h1": "Shopify competitor analysis: how to validate a new market with Apify and Claude",
  "canonical": "https://blog.apify.com/shopify-competitor-analysis/",
  "ogTitle": "Thinking of selling your Shopify products abroad?",
  "ogImage": "https://storage.ghost.io/c/.../Shopify-competitor-analysis.png",
  "metaRobots": "index,follow",
  "isNoindex": "false",
  "wordCount": "4577",
  "error": "",
  "scrapedAt": "2026-11-10T02:03:01.520Z"
}

🌐 HTTP Status, Redirect & Performance Fields

Field	Type	Coverage	Description
url	string	🟢 High (100%)	The input URL checked (normalized with https:// when bare).
statusCode	string	🟢 High (~99%)	HTTP response status code ("200", "301", "404", "500"…; empty only on network/DNS failure).
ok	string	🟢 High (100%)	"true" when HTTP status is 200–399, "false" on 4xx/5xx or network error.
finalUrl	string	🟢 High (~99%)	Final destination URL after following HTTP redirects.
redirected	string	🟢 High (~99%)	"true" if finalUrl differs from url (ignoring trailing slash), otherwise "false".
responseTimeMs	string	🟢 High (~99%)	Total HTTP response time in milliseconds.
isHttps	string	🟢 High (~99%)	"true" when finalUrl uses HTTPS, "false" for HTTP.
contentType	string	🟢 High (~99%)	Primary MIME type from Content-Type header (text/html, application/pdf, image/png…).
contentLength	string	🟡 Conditional	Response body size in bytes when the server sends a Content-Length header.
server	string	🟡 Conditional	Web server / CDN identifier from the Server header (cloudflare, nginx, openresty…).
error	string	🟢 High (100%)	Empty string "" on reachable responses; contains the failure reason on DNS/timeout/reset errors.
scrapedAt	string	🟢 High (100%)	ISO 8601 timestamp when the URL was checked.

🔎 On-Page SEO & Content Fields (checkSeo: true)

Field	Type	Coverage	Description
title	string	🟢 High (~99% on HTML)	Text of the first <title> element.
metaDescription	string	🟢 High (~85% on HTML)	Content of <meta name="description">.
h1	string	🟢 High (~92% on HTML)	Text of the first <h1> heading on the page.
canonical	string	🟢 High (~88% on HTML)	href of <link rel="canonical">.
ogTitle / ogImage	string	🟢 High (~85% on HTML)	Open Graph og:title and og:image meta tag values.
metaRobots	string	🟡 Conditional	Content of <meta name="robots"> when declared on the page.
isNoindex	string	🟢 High (100% on HTML)	"true" if metaRobots contains noindex, otherwise "false".
wordCount	string	🟢 High (100% on HTML)	Total visible word count inside <body>.

---

🔁 Pagination, sitemaps & 1,000+ scaling

• Exact bulk list mode (maxItems: 0): Pass 100, 1,000 or 100,000 URLs (via pasted list, uploaded file, or requestsFromUrl pointing to a remote CSV/txt/XML sitemap) and the Actor checks every single URL with zero artificial caps.
• Seed expansion mode (maxItems > seed URLs): When maxItems is set higher than the number of direct URLs you provided (for example, 3 prefill URLs with maxItems: 50, or 1 homepage URL with maxItems: 1000), the Actor automatically discovers and checks <a href> links from the visited pages until maxItems unique URLs are audited.
• Empty-run demo (100 URLs): Running with {} checks 100 built-in reference URLs at high concurrency in ~15–25 seconds.

---

⚙️ How to use

1. Open Bulk URL Checker in the Apify Console and click Try for free.
2. Paste your URLs, upload a text/CSV file, or add a remote XML sitemap URL via requestsFromUrl.
3. Keep checkSeo: true for full SEO tag extraction (or set false for pure HTTP status speed), and adjust maxItems and maxConcurrency.
4. Click Start, then export your results as JSON, CSV, Excel, or read the Dataset / Key-Value Store OUTPUT via the Apify API.

---

🔌 API usage

Run asynchronously via REST API

curl -X POST "https://api.apify.com/v2/acts/haketa~bulk-url-checker/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      { "requestsFromUrl": "https://blog.apify.com/sitemap-posts.xml" }
    ],
    "checkSeo": true,
    "maxItems": 500,
    "maxConcurrency": 50
  }'

Run synchronously and get checked URLs immediately

curl -X POST "https://api.apify.com/v2/acts/haketa~bulk-url-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      { "url": "https://apify.com" },
      { "url": "https://crawlee.dev" },
      { "url": "https://github.com/apify" }
    ],
    "checkSeo": true,
    "maxItems": 3
  }'

---

Call from Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "urls": [
        {"url": "https://apify.com"},
        {"url": "https://docs.apify.com"},
        {"url": "https://crawlee.dev"},
    ],
    "checkSeo": True,
    "maxItems": 100,
    "maxConcurrency": 50,
    "timeoutSecs": 30,
}

run = client.actor("haketa/bulk-url-checker").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"[{item['statusCode']}] {item['url']} -> {item['finalUrl']} ({item['responseTimeMs']} ms) | Title: {item['title']}")

Call from Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('haketa/bulk-url-checker').call({
    urls: [
        { requestsFromUrl: 'https://blog.apify.com/sitemap-posts.xml' },
    ],
    checkSeo: true,
    maxItems: 500,
    maxConcurrency: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Audited ${items.length} URLs`);

---

🔄 Integrations & automated SEO / QA workflows

Connect Bulk URL Checker to your SEO monitoring, DevOps or RevOps pipelines:

Integration	Workflow example
🚨 Slack, PagerDuty & Email Alerts	Schedule hourly or daily runs against critical landing pages and trigger an instant alert via Make/Zapier/n8n whenever any URL returns ok: "false" or isNoindex: "true".
📊 Google Sheets & Looker Studio	Push weekly sitemap audit results (statusCode, redirected, responseTimeMs, title, metaDescription, h1, canonical, wordCount) to a live technical SEO dashboard.
🧹 Clay, HubSpot & Salesforce	Pre-validate thousands of prospect company domains before running expensive enrichment workflows — drop dead domains (ok: "false") and keep clean finalUrl domains.
🗄️ BigQuery, Snowflake & S3	Archive historical crawl snapshots across 10,000+ product or editorial URLs to track response latency, title/meta changes and redirect hygiene over time.
🤖 AI SEO & QA Agents	Invoke the Actor via Apify MCP or synchronous REST API so autonomous SEO and QA agents can verify live URLs and inspect on-page metadata in real time.

---

⚠️ Limitations & technical notes

• Server-rendered HTML (no JavaScript execution): For maximum speed and low cost at 1,000+ to million-URL scale, the Actor uses plain HTTP requests (CheerioCrawler) rather than a headless browser. It reads the initial server-returned HTML, which matches what lightweight crawlers and link validators see, but tags injected exclusively by client-side JavaScript after page load will not appear.
• Origin rate-limiting on single-domain runs: Checking 1,000+ URLs on a single small website at maxConcurrency: 200 can trigger 429 Too Many Requests or 503 responses from the origin firewall. Lower maxConcurrency to 10–30 (and optionally enable proxyConfiguration) when auditing a single domain, and save high concurrency (100–500) for multi-domain lists.
• Header-dependent fields: contentLength and server are populated whenever the target server sends Content-Length and Server HTTP response headers (chunked responses often omit Content-Length).

---

💡 Pro tips for bulk auditing

• Use requestsFromUrl for XML sitemaps and large CSVs — paste any https://.../sitemap.xml URL into the URLs source editor (From URL) and Crawlee will automatically extract and check every <loc> entry.
• Turn off checkSeo (false) when cleaning 10,000+ sales lead domains — skipping HTML DOM parsing maximizes throughput and reduces memory usage.
• Lower maxConcurrency (e.g. 10–25) when auditing a single small server so you don't trigger origin rate-limiting; raise it to 100–300 when checking thousands of distinct domains.
• Read SUMMARY from Key-Value Store for instant health counts (checked, okCount, redirectedCount, errorCount).

---

❓ FAQ

How many URLs can I check in one run? From a few URLs up to millions. Set maxItems: 0 to check every URL in your input list or sitemap without limit, or set maxItems: 1000+ on seed URLs to automatically discover and check internal/outbound links up to your target.

What happens when a URL returns 404, 500 or times out? It is always included in the output dataset with ok: "false", its HTTP statusCode (if received), and a clean error message — never silently dropped.

Does it work on PDFs, images and XML feeds? Yes. All MIME types (*/*) are accepted and reported with their real statusCode, contentType, contentLength and responseTimeMs. SEO tag fields are parsed only on HTML responses.

Do I need proxies? No for most runs — direct HTTP requests are used by default so runs stay fast and cheap. Enable Apify Proxy only if you are auditing a single rate-limited host at high volume or testing geo-specific redirects.

How does seed link discovery work when maxItems is set higher than my input URLs? When you provide a small seed list (for example, 1–3 homepage URLs) and set maxItems to a larger number (such as 50 or 1,000), the Actor checks your initial URLs first and automatically extracts, normalizes and enqueues <a href> links found on those pages until maxItems unique URLs have been checked. When maxItems: 0 (the default), link discovery is disabled and the Actor checks only the exact URLs in your input list or sitemap.

Where can I find a quick summary of healthy vs. broken URLs after a run? In addition to the full Dataset, every run saves a SUMMARY JSON object (checked, okCount, redirectedCount, errorCount, finishedAt) and an OUTPUT array to the run's default Key-Value Store, which you can fetch in a single API call.

Are bare domains without https:// supported? Yes. Entries like apify.com or github.com/apify/crawlee are automatically normalized to https://apify.com and https://github.com/apify/crawlee before checking, and any http:// -> https:// or non-www -> www redirect is captured in redirected and finalUrl.

Sample bulk URL auditing playbooks

Playbook	Recommended input configuration
🗺️ Full XML Sitemap Technical SEO Audit	urls: [{ "requestsFromUrl": "https://yoursite.com/sitemap.xml" }], checkSeo: true, maxItems: 0, maxConcurrency: 40
🧹 10,000+ B2B Prospect Domain Health Check	urls: uploaded CSV/TXT of domains, checkSeo: false, maxItems: 0, maxConcurrency: 200, timeoutSecs: 15
🔀 Domain Migration & Redirect Chain Verification	urls: legacy URLs list, checkSeo: true, maxItems: 0, maxConcurrency: 50 — filter output on redirected and finalUrl
🕷️ Seed-URL 1,000+ Link Discovery & Audit	urls: [{ "url": "https://yoursite.com" }], checkSeo: true, maxItems: 1000, maxConcurrency: 50

---

🔗 Related scrapers

• Bulk Metadata Extractor — Titles, Meta & Open Graph
• OutSEO Scraper — SEO Keyword & Competitor Intelligence
• Google Trends Scraper — Search Interest & Trending Topics
• Google Play Scraper — Android App Store Metadata & Reviews
• App Store Scraper — iOS App Metadata & Ratings

---

📋 Changelog

• v1.2 (2026-11) — Expanded built-in empty-run demo to 100 diverse URLs, added automatic on-page link discovery when maxItems exceeds seed URLs (supporting 50-item prefill and 1,000+ URL expansion), added strict URL normalization/deduplication (0 duplicates), and added Key-Value Store OUTPUT and SUMMARY records.
• v1.1 (2026-11) — Added additionalMimeTypes: ['*/*'] so non-HTML assets (PDFs, images, XML feeds) report real HTTP status and contentLength, and verified 504-URL sitemap benchmark.
• v1.0 (2026-10) — Initial release with high-concurrency HTTP status checking, redirect resolution, timing metrics and on-page SEO tag extraction.