Bulk Metadata, Open Graph & JSON-LD Extractor — SEO Tags

Bulk Metadata, Open Graph & JSON-LD Extractor — SEO Tags at Scale

Extract on-page SEO metadata, Open Graph, Twitter Card and schema.org JSON-LD from thousands of URLs at once. This metadata scraper reads the <head> of any webpage and returns, for each URL: title, meta description, canonical, Open Graph & Twitter Card tags, favicon, hreflang, RSS feed and structured data — no login, no coding, no API key. Paste a list of URLs and get a clean, comparable metadata inventory back. Whether you want to scrape website metadata for a handful of pages or audit an entire enterprise sitemap, this bulk metadata scraper does it in one run.

Export every result to JSON, CSV, Excel, or API.

Built for technical SEO specialists, content auditors, link-preview / unfurl developers, growth marketers, and data engineers enriching URL datasets who need the metadata layer of the web in bulk.

---

⚡ At a glance

Feature	Details
Data Extracted	Title, meta description, canonical, Open Graph (7 tags), Twitter Cards (6 tags), schema.org JSON-LD, hreflang, favicon, manifest
JSON-LD Schema Scope	Recursive extraction across all @type and @graph nodes (Product, Article, Organization, BreadcrumbList, etc.)
Throughput & Speed	High-concurrency Cheerio pool processing 500 URLs in ~3 minutes
Scale & Capacity	Supports individual page queries up to 100,000+ URL lists in a single job
Engine Architecture	Pure-HTTP lightweight pipeline bypassing browser rendering for maximum speed and cost efficiency
Output Formats	Clean datasets in JSON, CSV, Excel, XML, HTML table, and JSON Lines
Integrations	Apify REST API, webhooks, Python SDK, Node.js SDK, Make, Zapier, Google Sheets
Proxy Flexibility	Keyless direct HTTP by default; optional Apify Proxy for rate-limited corporate targets

---

🔎 Why scrape website metadata?

Every webpage's HTML <head> encapsulates critical signals that dictate how that URL behaves across search engines, social platforms, and messaging applications. The <title>, <meta name="description">, and <link rel="canonical"> define search ranking and SERP click-through rates. The Open Graph (og:*) and Twitter Card (twitter:*) tags govern how links unfurl in Slack, LinkedIn, Twitter/X, and WhatsApp. Furthermore, schema.org JSON-LD structured data instructs Google and AI search engines on rich snippet eligibility (Product, FAQPage, Review, Recipe, Organization).

Auditing these elements manually via browser developer tools or URL-by-URL browser extensions is painfully slow and impossible to scale across sitemaps or competitor catalogs.

This bulk metadata scraper automates the entire audit process. Paste a list of URLs or bare domains, and the extractor fetches each page over lightweight HTTP, parses <head> markup with Cheerio, resolves relative URLs to absolute links, walks JSON-LD schema graphs recursively, and outputs a complete, unified metadata audit record for each URL.

Common reasons teams scrape website metadata:

• Technical SEO audits — identify missing meta titles, truncated descriptions, duplicate canonicals, and missing meta robots across entire sites.
• Structured data & rich snippet audits — verify which product, blog, or category pages contain valid schema.org markup and which lack structured data.
• Social link preview & unfurling systems — power chat apps, CMS link unfurlers, and bookmark managers with real-time Open Graph titles and preview images.
• Site migration & redesign QA — capture metadata snapshots pre- and post-migration to verify zero loss of canonicals or title tags.
• Competitor intelligence — analyze how rival publications or e-commerce stores optimize their meta descriptions, OG images, and schema entities.
• Lead & CRM enrichment — enrich raw landing page URLs with company branding, favicons, site names, and descriptive titles.

---

✨ What this website metadata scraper does

• ✅ Reads any list of URLs in bulk — from a quick test of 10 links to 100,000+ URLs fanned out across parallel workers.
• ✅ Returns one clean record per URL with 37 normalized fields ready for immediate analysis.
• ✅ Standard SEO tags — title, metaDescription, canonical, robots, viewport, themeColor, author, keywords, HTML lang, and charset.
• ✅ Open Graph metadata — ogTitle, ogDescription, ogImage, ogType, ogSiteName, ogUrl, ogLocale.
• ✅ Twitter Card metadata — twitterCard, twitterTitle, twitterDescription, twitterImage, twitterSite, twitterCreator.
• ✅ Icons, feeds & manifests — favicon, appleTouchIcon, web app manifestUrl, and RSS/Atom feedUrl.
• ✅ Internationalization tracking — extracts all alternate language versions from link[rel=alternate][hreflang].
• ✅ Deep schema.org JSON-LD parsing — extracts all script[type="application/ld+json"] blocks, recursively indexes every @type (including nested @graph objects), and optionally exports raw JSON.
• ✅ Diagnostic response telemetry — records statusCode, final redirected finalUrl, MIME contentType, and detailed error messages for broken pages.
• ✅ Fully resolved absolute URLs — automatically converts relative favicon, image, and canonical paths into valid absolute URLs.
• ✅ Resilient retry engine — handles transient 429 and 5xx errors with backoff retries and connection recycling.

---

🎯 Use cases & workflows

1. Site-Wide Technical SEO Audit

Export your sitemap XML URLs and feed them into this Actor. Export the results to CSV and filter for:

• title == "" or title.length > 60
• metaDescription == "" or metaDescription.length > 160
• canonical != finalUrl (identifies improper canonicalization loops)

2. Rich Snippet Schema Opportunity Analysis

Filter results for schemaTypes == "" to identify high-traffic e-commerce or editorial pages missing schema.org Product or Article tags that could qualify for enhanced Google search snippets.

3. Automated Link-Preview Generation

Call the Actor via API whenever an internal user bookmarks or submits an external URL. Store the returned ogTitle, ogDescription, and ogImage to generate rich preview cards.

4. Migration Regression Testing

Run an extraction on your staging environment before launching a site rebuild. Diff staging metadata against production metadata to ensure titles, descriptions, and hreflang tags remain intact.

---

🧩 Schema.org Structured Data Entity Reference

This Actor includes an advanced recursive JSON-LD parser that identifies schema.org entities even when nested multiple levels deep within @graph structures:

Schema Entity	Typical URL Types	Key Attributes Monitored	Rich Result Eligibility
Product	E-commerce PDPs, SaaS pricing	name, offers, aggregateRating, brand, sku	Product snippets, price & stock badges
Article / NewsArticle	Editorial posts, news articles	headline, author, datePublished, publisher	Top Stories, author rich carousels
BreadcrumbList	Category and product taxonomies	itemListElement, position, name, item	Breadcrumb hierarchy in search snippets
Organization	Homepages, about pages, contact	name, url, logo, contactPoint, sameAs	Knowledge Graph brand panels
WebSite	Domain homepages	name, url, potentialAction (SearchAction)	Google Sitelinks Search Box
FAQPage	Help centers, product FAQs	mainEntity (Question, acceptedAnswer)	Expandable FAQ search accordions
Review	Testimonial pages, merchant hubs	itemReviewed, reviewRating, author	Star rating review snippets
HowTo	Tutorial and DIY documentation	step, supply, tool, totalTime	HowTo visual rich cards
LocalBusiness	Storefronts, branch directories	address, telephone, openingHoursSpecification	Local 3-Pack and Maps integration
JobPosting	Careers pages, job boards	title, baseSalary, datePosted, employmentType	Google for Jobs listings

---

🔄 End-to-End Automation & Webhook Integration

Connect this Actor directly to automated marketing and CRM operations using Apify Webhooks and modern iPaaS tools:

Automated Sitemap Audit with Make (Integromat) or n8n

1. Trigger on Run Succeeded: Create an Apify webhook that activates whenever bulk-metadata-extractor completes.
2. Fetch Dataset Items: In Make or n8n, iterate through the resulting dataset records.
3. Filter Missing Critical Tags: Identify records where title == "" OR metaDescription == "" OR canonical == "".
4. Export to Jira / Asana: Automatically generate technical SEO bug tickets for engineering or content teams with the exact URL and missing tags.
5. Slack Alert Notification: Post a summary message to #seo-alerts displaying the total audit count and missing tag percentages.

Sample CSV Export Structure

When exporting results to CSV or spreadsheet applications, the data formats into predictable columns:

url,finalUrl,statusCode,contentType,title,metaDescription,canonical,lang,charset,robots,viewport,ogTitle,ogDescription,ogImage,schemaTypes,jsonLdCount,error,scrapedAt
"https://kubernetes.io/","https://kubernetes.io/","200","text/html","Kubernetes","Production-Grade Container Orchestration","https://kubernetes.io/","en","utf-8","index, follow","width=device-width","Kubernetes","Open source container system...","https://kubernetes.io/images/k8s.png","Organization, SearchAction","1","","2026-11-10T16:04:00.000Z"

---

📥 Input parameters

Configure your run via the Apify Console or pass a JSON payload directly through the API. Every field is optional.

Field	Type	Default	Description
urls	array (stringList)	[]	List of webpage URLs or bare domains (e.g., theverge.com). Leave empty for a popular demo sample.
includeJsonLd	boolean	true	Include the full schema.org JSON-LD string in structuredData. Set to false for a lighter output containing only schemaTypes.
maxItems	integer	0	Maximum URLs to process. 0 = unlimited (processes your entire list). Supports 1,000+ to 100K+ URLs.
maxConcurrency	integer	20	Parallel concurrent requests (1–100). Raise for large audits on high-bandwidth networks.
proxyConfiguration	object	{ "useApifyProxy": false }	Optional Apify Proxy settings for targets with aggressive anti-scraping firewalls.

Example input — Targeted multi-site audit

{
  "urls": [
    "https://www.theverge.com",
    "https://github.com",
    "https://stripe.com",
    "https://www.nytimes.com"
  ],
  "includeJsonLd": true,
  "maxConcurrency": 20
}

Example input — 1,000+ large scale sitemap audit

{
  "urls": [
    "https://example.com/page-1",
    "https://example.com/page-2",
    "https://example.com/page-1000"
  ],
  "includeJsonLd": false,
  "maxItems": 1500,
  "maxConcurrency": 50
}

---

📤 Output structure

Each URL generates a comprehensive record containing SEO, social, and structured data attributes:

{
  "url": "https://kubernetes.io/",
  "finalUrl": "https://kubernetes.io/",
  "statusCode": "200",
  "contentType": "text/html",
  "title": "Kubernetes",
  "metaDescription": "Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications.",
  "canonical": "https://kubernetes.io/",
  "lang": "en",
  "charset": "utf-8",
  "robots": "index, follow",
  "viewport": "width=device-width,initial-scale=1,shrink-to-fit=no",
  "themeColor": "#326de6",
  "ogTitle": "Production-Grade Container Orchestration",
  "ogDescription": "Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications.",
  "ogImage": "https://kubernetes.io/images/kubernetes-open-graph.png",
  "ogType": "website",
  "ogSiteName": "Kubernetes",
  "ogUrl": "https://kubernetes.io/",
  "ogLocale": "en",
  "twitterCard": "summary",
  "twitterTitle": "Production-Grade Container Orchestration",
  "twitterDescription": "Kubernetes, also known as K8s, is an open source system...",
  "twitterImage": "https://kubernetes.io/images/kubernetes-open-graph.png",
  "favicon": "https://kubernetes.io/images/kubernetes.png",
  "appleTouchIcon": "https://kubernetes.io/images/favicon.png",
  "manifestUrl": "https://kubernetes.io/manifest.json",
  "feedUrl": "https://kubernetes.io/feed.xml",
  "hreflang": "bn, zh-cn, fr, de, hi, id, it, ja, ko, fa, pl, pt-br, ru, es, uk, vi",
  "schemaTypes": "Organization, SearchAction",
  "jsonLdCount": "1",
  "structuredData": "[{\"@context\":\"https://schema.org\",\"@type\":\"Organization\",\"url\":\"https://kubernetes.io\"}]",
  "error": "",
  "scrapedAt": "2026-11-10T16:04:00.000Z"
}

---

📊 Extracted fields & data coverage

Field	Coverage	Type	Description
🔗 url	🟢 High (100%)	String	Original input URL submitted for extraction
📍 finalUrl	🟢 High (~99%)	String	Final URL resolved after HTTP redirects
📡 statusCode	🟢 High (100%)	String	HTTP response code (e.g., 200, 301, 404, 500)
📄 contentType	🟢 High (~99%)	String	MIME type header (e.g., text/html, application/pdf)
🏷️ title	🟢 High (~98%)	String	Document <title> tag (falls back to og:title)
📝 metaDescription	🟢 High (~88%)	String	meta[name="description"] text content
🎯 canonical	🟢 High (~95%)	String	link[rel="canonical"] URL resolved to absolute form
🌐 lang	🟢 High (~90%)	String	HTML document language code (e.g., en, es, fr)
🔤 charset	🟢 High (~92%)	String	Document character encoding (e.g., utf-8)
🤖 robots	🟡 Conditional (~40%)	String	Meta robots directive (e.g., index, follow, noindex)
📱 viewport	🟢 High (~95%)	String	Mobile responsive viewport meta tag
🎨 themeColor	🟡 Conditional (~30%)	String	Mobile browser toolbar theme color
🖼️ ogTitle	🟢 High (~85%)	String	Open Graph social sharing title
📄 ogDescription	🟢 High (~82%)	String	Open Graph preview card description
🌅 ogImage	🟢 High (~80%)	String	Open Graph social thumbnail image URL
🏷️ ogType	🟢 High (~78%)	String	Open Graph object type (website, article, product)
🏢 ogSiteName	🟢 High (~75%)	String	Brand or website name declared via Open Graph
🐦 twitterCard	🟢 High (~70%)	String	Twitter Card format (summary, summary_large_image)
🔖 favicon	🟢 High (~99%)	String	Website favicon URL resolved to absolute path
🌍 hreflang	🟡 Conditional (~35%)	String	Comma-separated alternate localized language codes
🧩 schemaTypes	🟢 High (~65%)	String	Detected schema.org @type entities (comma-separated)
🔢 jsonLdCount	🟢 High (100%)	String	Number of valid JSON-LD script blocks identified
📦 structuredData	🟡 Conditional (~65%)	String	Serialized raw JSON-LD payload (when enabled)
⚠️ error	⚪ Optional	String	Diagnostic message if the page failed to load
⏰ scrapedAt	🟢 High (100%)	String	ISO 8601 UTC timestamp of execution

---

🔁 Scale & 1,000+ deep parallel processing

This Actor is engineered for enterprise-grade bulk throughput:

• 1,000+ to 100K+ URLs per run: Submit massive URL files or sitemaps without running out of memory.
• Configurable worker concurrency: Adjust maxConcurrency between 1 and 100 to balance speed and target server politeness.
• Zero browser rendering overhead: Bypassing Chromium eliminates massive RAM and CPU bottlenecks, cutting runtime costs by up to 90%.
• De-duplicated input processing: The Actor automatically de-duplicates input URL strings to prevent redundant network requests.

---

💻 Developer integration & code samples

Python SDK

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "urls": [
        "https://www.theverge.com",
        "https://stripe.com",
        "https://github.com"
    ],
    "includeJsonLd": True,
    "maxConcurrency": 20
}

run = client.actor("haketa/bulk-metadata-extractor").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"URL: {item['url']} | Title: {item['title']} | Schema: {item['schemaTypes']}")

Node.js SDK

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('haketa/bulk-metadata-extractor').call({
    urls: ['https://stripe.com', 'https://github.com'],
    includeJsonLd: false,
    maxConcurrency: 20,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.table(items.map(i => ({ url: i.url, title: i.title, ogImage: i.ogImage, schema: i.schemaTypes })));

cURL API

curl -X POST "https://api.apify.com/v2/acts/haketa~bulk-metadata-extractor/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://www.theverge.com", "https://stripe.com"],
    "includeJsonLd": true
  }'

---

💡 Practical tips for optimal metadata extraction

1. Bare domains are supported: You can submit theverge.com or stripe.com; the engine automatically prepends https://.
2. Exclude raw JSON-LD for speed: If you only need schema entity names (schemaTypes) and not raw JSON blocks, set includeJsonLd: false to reduce dataset size.
3. Audit hreflang consistency: Inspect the hreflang column to verify that regional sites correctly declare cross-border alternates.
4. Inspect contentType: Check contentType to confirm that your URLs returned standard text/html rather than redirected API responses or PDF documents.

---

❓ Frequently Asked Questions

Does the scraper render client-side JavaScript?
No. It inspects server-rendered HTML markup where standard <head> tags and JSON-LD schema blocks are defined. This ensures ultra-fast execution and low operating costs.

Can I analyze 20,000 URLs at once?
Yes. You can paste thousands of URLs into urls. The engine processes requests concurrently through an internal worker pool.

Are relative image and favicon paths resolved?
Yes. All relative links for og:image, twitter:image, canonical, and favicon are converted to fully qualified absolute HTTP(S) URLs.

What formats can I download?
Results can be downloaded in JSON, CSV, Excel, XML, HTML table, or JSON Lines via the Apify Console or API.

How does error handling work?
If a target URL returns a 404, 403, or connection timeout, the Actor still outputs a record containing the statusCode and descriptive error message.

Can I use this Actor to extract metadata for private intranet pages?
If your intranet or staging pages require specific HTTP authentication headers or are accessible only via a dedicated corporate proxy, configure your proxy credentials in proxyConfiguration to route traffic securely.

How does the extractor handle multiple JSON-LD script tags on a single page?
Many modern CMS systems output separate JSON-LD tags for breadcrumbs, products, and organization info. The Actor parses every script tag on the page, combines unique entity types into schemaTypes, and consolidates all raw objects into structuredData.

Does this scraper extract Open Graph audio or video tags?
The Actor focuses on the core Open Graph properties (title, description, image, type, site_name, url, locale) that power the vast majority of social cards and search engines.

Is it safe to run across large corporate sites without causing server overload?
Yes. You can regulate the load on target servers using the maxConcurrency parameter (e.g. setting it to 5 or 10 for polite, throttled crawls).

---

🔗 Related SEO & data extraction scrapers

• Bulk Tech Detector — detect website software stacks, CMS, frameworks, and analytics in bulk.
• Sanctions & Watchlist Screening Scraper — screen names and corporate entities against OFAC, EU, UN, and UK regimes.
• Treasury & Sovereign Debt Data Scraper — extract official US Treasury yield curves, debt metrics, and exchange rates.
• Nevada Contractor License Scraper — extract licensed contractors, bonds, and phone contacts from the NSCB registry.

---

📋 Changelog

v1.2 (2026-11)

• Unlocked 1,000+ to 100K+ batch processing: confirmed unrestricted queue execution when maxItems: 0 is set.
• Enhanced input normalization: added robust handling for { url: "..." } object inputs and bare domain strings.
• Upgraded documentation: added visual badges, ⚡ At a glance quick reference, field coverage tables, and Python/Node.js SDK examples.

v1.1 (2026-07)

• Added recursive schema.org JSON-LD parsing across nested @graph entities.
• Added absolute URL resolution for ogUrl and Twitter card images.

v1.0 (2026-06)

• Initial release with keyless Cheerio metadata extraction.