# Website Metadata Extractor(sitemap, socialLinks, robotsTxt) (`codescraper/website-metadata-extractor`) Actor

A fast metadata extractor for all meta tags, robots.txt, sitemaps, 15+ social links, H1s, word count, JSON-LD data, contacts & technology detection. Now with AI business & industry classification. Pay-per-event from $1/1k domains with zero server rental.

- **URL**: https://apify.com/codescraper/website-metadata-extractor.md
- **Developed by:** [CodeScraper](https://apify.com/codescraper) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 120 total users, 6 monthly users, 97.7% runs succeeded, 3 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $0.70 / 1,000 domain analyzed & extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🌐 Website Metadata Extractor (Sitemaps, Social Links, Robots.txt & AI Industry Classifier)

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-orange.svg)](https://apify.com)
[![Pricing](https://img.shields.io/badge/Pricing-Pay--Per--Event%20\($1.50%2F1k%20domains\)-green.svg)](https://apify.com)
[![AI Powered](https://img.shields.io/badge/AI-Groq%20openai%2Fgpt--oss--120b-purple.svg)](https://groq.com)
[![Status](https://img.shields.io/badge/Status-Active%20&%20Maintained-blue.svg)](https://apify.com)
[![Speed](https://img.shields.io/badge/Speed-High--Concurrency%20Cheerio-brightgreen.svg)](https://crawlee.dev)

A high-performance, enterprise-grade **Apify Actor** that extracts comprehensive **website metadata, SEO tags, robots.txt rules, sitemap files, verified social media profiles, Open Graph & Twitter Cards, technology stacks, JSON-LD schemas, content headings, direct contact details, and AI-powered business industry classification** from any list of domains or URLs.

Built for **SEO professionals, data engineers, marketing agencies, growth hackers, and competitive intelligence teams**, it delivers deep website audits in milliseconds with zero browser overhead at a fraction of the cost of headless scrapers.

***

### ⚡ What’s New in Version 2.0?

- 🤖 **AI Business & Company Intelligence (`natureOfBusiness`):** Integrated with Groq’s high-speed `openai/gpt-oss-120b` model to accurately extract complete company intelligence: 100+ standard industry categories, 1-sentence executive summary, B2B vs. B2C model, business & revenue model, ideal customer profile (target audience), core offerings & products, estimated company stage/scale, and compliance/trust signals (SOC2, GDPR, HIPAA, etc.).
- 💳 **Transparent Pay-Per-Event (PPE) Monetization:** Pay only for successful domains. Unreachable, invalid, or dead domains cost **$0.00**.
  - **Standard Extraction:** $1.50 per 1,000 domains ($1.00 domain analyzed + $0.50 dataset result).
  - **With AI Classification Enabled:** $4.50 per 1,000 domains ($1.00 domain analyzed + $0.50 dataset result + $3.00 AI intelligence).
- 📱 **Verified Social Links Extraction:** Multi-source detection covering **15+ major platforms** (Facebook, Twitter/X, LinkedIn, Instagram, YouTube, TikTok, Pinterest, Reddit, GitHub, Discord, Telegram, WhatsApp, Snapchat, Threads, Medium) with intelligent noise and tracking parameter filtering.
- 🤖 **Structured Robots.txt & Sitemap Engine:** Intelligently parses user-agent rules (Allow, Disallow, Crawl-Delay) and discovers XML sitemaps without hanging on bot protections or WAFs.
- 🧱 **30+ Modern Technology Fingerprints:** Detects CMS platforms (WordPress, Shopify, Wix, Squarespace, Webflow), modern frameworks (Next.js, React, Vue, Nuxt, Angular, Astro), analytics (GA4, GTM, Clarity, Hotjar), pixels, and CDNs (Cloudflare, Fastly).
- 🏢 **Company Logo & Legal Name:** Parses Schema.org Organization and Corporation schemas to extract high-res logos and official legal entity names.
- ⚡ **Ultra-Fast AI Processing:** High-speed company intelligence inference with intelligent rate optimization and high throughput.
- 🌐 **Universal URL Flexibility:** Accepts bare domains (`apify.com`), subdomain prefixes (`www.apify.com`), or full URLs (`https://...`). Output provides both original and normalized URLs.

***

### 🚀 Key Features & Extracted Data

| Category | Extracted Data Fields |
| :--- | :--- |
| **🤖 AI Nature of Business** | Industry Classification (100+ standard sectors), 1-Sentence Company Summary, B2B vs. B2C Model, Business Model, Target Audience (ICP), Core Offerings, Company Stage, Compliance Signals |
| **🏢 Company & Branding** | Official Legal Name, Company Logo URL, Favicon, Apple Touch Icon |
| **Core SEO Tags** | Page Title, Meta Description, Canonical URL, Charset, Language, Theme Color |
| **Robots & Sitemaps** | Full `robots.txt` rules parsed by User-Agent (Allow / Disallow / Crawl-Delay), Validated Sitemap XML URLs, RSS / Atom Feeds |
| **15+ Social Profiles** | Facebook, Twitter / X, LinkedIn, Instagram, YouTube, TikTok, Pinterest, Reddit, GitHub, Discord, Telegram, WhatsApp, Snapchat, Threads, Medium |
| **Contact Information** | Direct email addresses (`mailto:`) and phone numbers (`tel:`) *(optional)* |
| **Open Graph (OG)** | `og:title`, `og:description`, `og:image`, `og:url`, `og:type`, `og:site_name`, `og:locale`, and all custom OG attributes |
| **Twitter Cards** | `twitter:card`, `twitter:site`, `twitter:creator`, `twitter:title`, `twitter:description`, `twitter:image` |
| **Structured Data** | Full Schema.org JSON-LD objects (`Organization`, `WebSite`, `Product`, `Article`, `FAQPage`, `LocalBusiness`, etc.) |
| **Headings & Hierarchy** | Primary `H1`, All `H1` tags, All `H2` tags, All `H3` tags |
| **Content Metrics** | Total Word Count and Total Character Count of body text |
| **Site Verification** | Google Site Verification, Bing / Microsoft, Yandex, Baidu, Pinterest, and Facebook Domain Verification |
| **International SEO** | Hreflang alternate language and regional URL mapping |
| **Technology Stack** | CMS, E-Commerce platforms, JS frameworks, Analytics tools, Pixels, and CDNs |

***

### 💰 Transparent Pay-Per-Event Pricing

This Actor utilizes Apify’s official **Pay-Per-Event (PPE)** pricing model. You only pay for what you extract, with zero monthly rental fees.

#### Pricing Events & Tiers

| Pricing Event | Event Code | Free Plan (No discount) | Starter Plan (Bronze discount) | Scale Plan (Silver discount) | Business Plan (Gold discount) |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Domain Analyzed & Extracted** | `domain-processed` | **$1.00** / 1,000 | **$0.90** / 1,000 | **$0.80** / 1,000 | **$0.70** / 1,000 |
| **Result Pushed** | `apify-default-dataset-item` | **$0.50** / 1,000 | **$0.50** / 1,000 | **$0.50** / 1,000 | **$0.50** / 1,000 |
| **AI Business Classification** *(Add-On)* | `ai-nature-of-business` | **$3.00** / 1,000 | **$2.80** / 1,000 | **$2.70** / 1,000 | **$2.60** / 1,000 |
| **Actor Start** *(Infrequent)* | `apify-actor-start` | **$0.005** | **$0.005** | **$0.005** | **$0.005** |
| **Platform Usage (Compute Units)** | — | **FREE** (Included) | **FREE** (Included) | **FREE** (Included) | **FREE** (Included) |

##### Effective Combined Totals:

- **Standard Metadata Extraction (AI Off):** **$1.50** / 1,000 *(Starter: $1.40 | Scale: $1.30 | Business: $1.20)*
- **AI-Enriched Extraction (AI On):** **$4.50** / 1,000 *(Starter: $4.20 | Scale: $4.00 | Business: $3.80)*

> 🛡️ **Zero-Cost Guarantee on Failed Domains:** If a website is down, blocked, or has a dead DNS record, you are charged **$0.00** for that domain. You only pay for successfully extracted results.

***

#### 💵 Cost Calculation Examples

##### Option A: Standard Metadata Extraction (AI Disabled — $1.50 / 1,000 domains)

- **1,000 Domains:** $0.005 (start) + (1 × $1.50) = **$1.50** *(Starter: $1.40 | Scale: $1.30 | Business: $1.20)*
- **5,000 Domains:** $0.005 + (5 × $1.50) = **$7.50** *(Starter: $7.00 | Scale: $6.50 | Business: $6.00)*
- **10,000 Domains:** $0.005 + (10 × $1.50) = **$15.00** *(Starter: $14.00 | Scale: $13.00 | Business: $12.00)*
- **50,000 Domains:** $0.005 + (50 × $1.50) = **$75.00** *(Starter: $70.00 | Scale: $65.00 | Business: $60.00)*

##### Option B: AI-Enriched Extraction with Nature of Business ($4.50 / 1,000 domains)

- **1,000 Domains:** $0.005 (start) + (1 × $4.50) = **$4.50** *(Starter: $4.20 | Scale: $4.00 | Business: $3.80)*
- **5,000 Domains:** $0.005 + (5 × $4.50) = **$22.50** *(Starter: $21.00 | Scale: $20.00 | Business: $19.00)*
- **10,000 Domains:** $0.005 + (10 × $4.50) = **$45.00** *(Starter: $42.00 | Scale: $40.00 | Business: $38.00)*
- **50,000 Domains:** $0.005 + (50 × $4.50) = **$225.00** *(Starter: $210.00 | Scale: $200.00 | Business: $190.00)*

***

### 🤖 AI Company Intelligence & Scalability

The AI classification is powered by Groq’s ultra-low latency inference engine using the high-performance **`openai/gpt-oss-120b`** model:

#### High-Throughput & Enterprise Scale

- **Ultra-Fast Inference:** Delivers complete company profiling in milliseconds per domain.
- **Proprietary Smart Context Distillation:** Intelligent semantic pre-filtering removes website noise, boilerplate navigation, and cookie banners to focus exclusively on core business signals.
- **Massive Concurrency:** Capable of analyzing up to 1,000 domains per minute smoothly without rate bottlenecks.
- **Built-in Resilience:** Enterprise-grade automatic retry mechanisms with dynamic backoff guarantee uninterrupted execution even under large batch processing loads.

#### Configuring Your Groq API Key

You can provide your Groq API key in two ways:

- **Option 1 (Recommended for Automation):** In the Apify Console under **Actor Settings → Environment Variables**, add `GROQ_API_KEY = your_key_here`.
- **Option 2:** Provide it directly in the Actor input field `groqApiKey` (stored securely as a secret).

***

### ⚙️ Input Parameters

| Parameter | Type | Default | Description |
| :--- | :---: | :---: | :--- |
| `startUrls` | Array | `["https://apify.com"]` | **(Required)** List of website URLs or domains. Accepts `domain.com`, `www.domain.com`, `http://`, or `https://`. |
| `enableAiClassification` | Boolean | `false` | **AI Nature of Business:** Enables industry categorization, 1-sentence summary, and B2B/B2C detection using Groq AI. |
| `groqApiKey` | String | `""` | Optional: Your Groq API key. If omitted, uses the `GROQ_API_KEY` Apify environment variable. |
| `customIndustries` | Array | `[]` | Optional: Custom list of industries for AI classification. If empty, the default 100+ global industries list is used. |
| `disableDomainAnalysis` | Boolean | `false` | **Fast Mode:** When `true`, skips fetching `robots.txt` and `sitemap.xml` for maximum crawl speed. |
| `maxConcurrency` | Integer | `20` | Maximum parallel domain requests (1 to 50). Recommended `20` for balanced speed and stability. |
| `extractMetaTags` | Boolean | `true` | Extract page title, meta description, Open Graph, Twitter Cards, and verification tags. |
| `extractRobotsTxt` | Boolean | `true` | Fetch and parse `robots.txt` rules by user-agent (Allow / Disallow / Crawl-Delay). |
| `extractSitemaps` | Boolean | `true` | Discover and validate sitemap XML files declared in robots.txt or at standard paths. |
| `extractSocialLinks` | Boolean | `true` | Extract verified social media profiles across 15+ networks. |
| `extractTechnologies` | Boolean | `true` | Detect CMS, JavaScript frameworks, e-commerce platforms, analytics, and CDNs. |
| `extractHeadings` | Boolean | `true` | Extract primary H1, all H1s, all H2s, and all H3 content headings. |
| `extractJsonLd` | Boolean | `true` | Extract and flatten Schema.org structured JSON-LD objects. |
| `extractHreflang` | Boolean | `true` | Extract international alternate language links (`hreflang`). |
| `extractContentMetrics` | Boolean | `true` | Count body text words and characters. |
| `includeContactInfo` | Boolean | `false` | Extract verified direct email addresses (`mailto:`) and phone numbers (`tel:`). |
| `socialPlatforms` | Array | *(All 15)* | Specific social platforms to target (e.g. `["linkedin", "twitter", "facebook"]`). |

***

### 🧩 Input Configurations

#### 1. Full-Featured Enterprise Audit + AI Nature of Business

```json
{
  "startUrls": [
    "apify.com",
    "https://shopify.com",
    "github.com",
    "https://stripe.com"
  ],
  "enableAiClassification": true,
  "disableDomainAnalysis": false,
  "maxConcurrency": 20,
  "extractMetaTags": true,
  "extractRobotsTxt": true,
  "extractSitemaps": true,
  "extractSocialLinks": true,
  "extractTechnologies": true,
  "extractHeadings": true,
  "extractJsonLd": true,
  "extractHreflang": true,
  "extractContentMetrics": true,
  "includeContactInfo": true,
  "socialPlatforms": [
    "facebook",
    "twitter",
    "linkedin",
    "instagram",
    "youtube",
    "tiktok",
    "pinterest",
    "reddit",
    "github",
    "discord",
    "telegram",
    "whatsapp",
    "snapchat",
    "threads",
    "medium"
  ]
}
```

#### 2. High-Speed On-Page SEO Mode (Fast Mode)

Skips external `robots.txt` and `sitemap.xml` network queries for ultra-fast on-page processing:

```json
{
  "startUrls": [
    "airbnb.com",
    "booking.com",
    "expedia.com"
  ],
  "disableDomainAnalysis": true,
  "extractRobotsTxt": false,
  "extractSitemaps": false,
  "maxConcurrency": 30
}
```

#### 3. B2B Lead Enrichment (Social Profiles + Contact Details + AI)

Optimized for lead qualification with company social links, contact info, and B2B classification:

```json
{
  "startUrls": [
    "https://hubspot.com",
    "https://salesforce.com",
    "https://notion.so"
  ],
  "enableAiClassification": true,
  "extractSocialLinks": true,
  "includeContactInfo": true,
  "socialPlatforms": [
    "linkedin",
    "twitter",
    "facebook",
    "youtube",
    "github"
  ],
  "extractMetaTags": true,
  "extractTechnologies": true,
  "disableDomainAnalysis": true
}
```

#### 4. Simple Minimalist Input

```json
{
  "startUrls": [
    "apify.com",
    "craigslist.org",
    "wikipedia.org"
  ]
}
```

***

### 📊 Output Example

Below is a complete output record showcasing the rich structured data extracted:

```json
{
  "url": "https://www.shopify.com/",
  "inputUrl": "shopify.com",
  "normalizedUrl": "https://shopify.com",
  "domain": "shopify.com",
  "brandName": "shopify",
  "statusCode": 200,
  "responseTimeMs": 348,
  "title": "Shopify: The All-in-One Commerce Platform for Businesses",
  "description": "Try Shopify free and start a business or grow an existing one. Get more than ecommerce software with tools to manage every part of your business.",
  "canonical": "https://www.shopify.com/",
  "favicon": "https://cdn.shopify.com/shopifycloud/web/assets/v1/favicon-default-6cbad9de243dbae3.ico",
  "appleTouchIcon": "https://cdn.shopify.com/b/shopify-brochure2-assets/c97c60ca19c64a8b5378d9f9e971f7bd.png",
  "logo": "https://cdn.shopify.com/b/shopify-brochure2-assets/logo.png",
  "legalName": "Shopify Inc.",
  "natureOfBusiness": {
    "industry": "Software as a Service (SaaS)",
    "summary": "Shopify provides an all-in-one cloud-based commerce platform and point-of-sale solutions for merchants worldwide.",
    "isB2B": true,
    "businessModel": "B2B SaaS / Subscription Commerce Platform",
    "targetAudience": [
      "E-commerce entrepreneurs and DTC brands",
      "Retail merchants migrating online",
      "Enterprise retailers needing headless checkout"
    ],
    "coreOfferings": [
      "Online store builder and hosted checkout",
      "Shopify POS hardware and software",
      "Shopify Payments and Merchant Solutions",
      "Enterprise headless commerce (Shopify Plus)"
    ],
    "estimatedStage": "Established Public Enterprise",
    "complianceSignals": [
      "PCI-DSS Level 1",
      "SOC2 Type II",
      "GDPR",
      "CCPA"
    ]
  },
  "themeColor": "#008060",
  "language": "en",
  "charset": "utf-8",
  "robotsTxt": {
    "userAgents": {
      "*": {
        "allow": [
          "/",
          "/pricing"
        ],
        "disallow": [
          "/admin",
          "/cart",
          "/checkout",
          "/checkouts/",
          "/orders",
          "/account"
        ],
        "crawlDelay": 1
      },
      "Googlebot": {
        "allow": [
          "/",
          "/products/"
        ],
        "disallow": [
          "/admin",
          "/cart"
        ]
      }
    }
  },
  "sitemapFileUrls": [
    "https://www.shopify.com/sitemap.xml",
    "https://www.shopify.com/sitemaps_list.xml"
  ],
  "feeds": [
    "https://www.shopify.com/blog.atom"
  ],
  "socialLinks": {
    "facebook": "https://facebook.com/shopify",
    "twitter": "https://x.com/shopify",
    "linkedin": "https://linkedin.com/company/shopify",
    "instagram": "https://instagram.com/shopify",
    "youtube": "https://youtube.com/shopify",
    "tiktok": "https://tiktok.com/@shopify",
    "pinterest": "https://pinterest.com/shopify",
    "reddit": "https://reddit.com/r/shopify",
    "github": "https://github.com/shopify",
    "discord": "https://discord.com/invite/shopify",
    "telegram": "https://t.me/shopifyofficial",
    "whatsapp": "https://wa.me/18005550199",
    "snapchat": "https://snapchat.com/add/shopify",
    "threads": "https://threads.net/@shopify",
    "medium": "https://medium.com/@shopify"
  },
  "contactInfo": {
    "emails": [
      "support@shopify.com",
      "press@shopify.com"
    ],
    "phones": [
      "+1-888-746-7439",
      "+1-855-837-7773"
    ]
  },
  "metaTags": {
    "viewport": "width=device-width, initial-scale=1",
    "robots": "index, follow",
    "author": "Shopify Inc.",
    "publisher": "https://www.shopify.com",
    "format-detection": "telephone=no"
  },
  "openGraph": {
    "title": "Shopify: The All-in-One Commerce Platform for Businesses",
    "description": "Try Shopify free and start a business or grow an existing one. Get tools to manage every part of your business.",
    "image": "https://cdn.shopify.com/b/shopify-brochure2-assets/ad99387e77223a56caca1c7b18e31970.png",
    "url": "https://www.shopify.com/",
    "type": "website",
    "siteName": "Shopify",
    "locale": "en_US"
  },
  "twitterCard": {
    "card": "summary_large_image",
    "site": "@Shopify",
    "creator": "@Shopify",
    "title": "Shopify: The All-in-One Commerce Platform",
    "description": "Start, run, and grow your business with Shopify ecommerce software.",
    "image": "https://cdn.shopify.com/b/shopify-brochure2-assets/ad99387e77223a56caca1c7b18e31970.png"
  },
  "securityVerification": {
    "googleSiteVerification": "g_verification_token_sample_8492049102",
    "bingSiteVerification": "BING_VERIFICATION_TOKEN_ABC123",
    "facebookDomainVerification": "fb_domain_verify_xyz789",
    "pinterestVerification": "pinterest_token_456"
  },
  "detectedTechnologies": [
    "Shopify",
    "React",
    "Next.js",
    "Ruby on Rails",
    "Google Analytics 4",
    "Google Tag Manager",
    "Cloudflare"
  ],
  "h1": "Be the next big thing",
  "allH1s": [
    "Be the next big thing"
  ],
  "allH2s": [
    "The one commerce platform behind it all",
    "There’s no better place for you to build",
    "Start selling in no time",
    "Move fast, grow faster"
  ],
  "allH3s": [
    "Customizable themes",
    "Built-in payment processing",
    "World-class checkout experience"
  ],
  "wordCount": 1420,
  "characterCount": 9850,
  "hreflangLinks": {
    "en": "https://www.shopify.com/",
    "en-gb": "https://www.shopify.com/uk",
    "de": "https://www.shopify.com/de",
    "fr": "https://www.shopify.com/fr",
    "es": "https://www.shopify.com/es",
    "ja": "https://www.shopify.com/jp"
  },
  "jsonLd": [
    {
      "@context": "https://schema.org/",
      "@type": "Corporation",
      "name": "Shopify",
      "url": "https://www.shopify.com/",
      "logo": "https://cdn.shopify.com/b/shopify-brochure2-assets/logo.png",
      "sameAs": [
        "https://facebook.com/shopify",
        "https://x.com/shopify",
        "https://linkedin.com/company/shopify",
        "https://instagram.com/shopify",
        "https://youtube.com/shopify",
        "https://github.com/shopify"
      ]
    }
  ],
  "scrapedAt": "2026-10-18T12:30:15.210Z"
}
```

#### Failed / Unreachable Domain Output ($0.00 Cost Guarantee)

When a domain has a DNS resolution failure, is permanently down, or refuses connections, a clean record is output with **zero charge**:

```json
{
  "url": "https://invalid-non-existent-domain-40918.com",
  "inputUrl": "invalid-non-existent-domain-40918.com",
  "normalizedUrl": "https://invalid-non-existent-domain-40918.com",
  "domain": "invalid-non-existent-domain-40918.com",
  "brandName": "invalid-non-existent-domain-40918",
  "statusCode": 500,
  "error": "Failed to load website: ENOTFOUND invalid-non-existent-domain-40918.com",
  "errorDetails": "getaddrinfo ENOTFOUND invalid-non-existent-domain-40918.com",
  "scrapedAt": "2026-10-18T12:30:18.420Z"
}
```

***

### 💡 Practical Use Cases

- 🔍 **Enterprise SEO Audits:** Audit meta descriptions, titles, robots.txt crawl rules, XML sitemaps, canonical tags, and H1/H2 content structures across thousands of domains in minutes.
- 💼 **B2B Lead Generation & Qualification:** Filter leads by confirmed B2B industry, company scale, verified LinkedIn handles, and direct contact details.
- 🧱 **Technology Stack Intelligence:** Detect adoption of Shopify, WordPress, Next.js, React, Google Analytics 4, and Cloudflare across entire industries.
- 🛡️ **Brand Protection & Competitor Tracking:** Track competitor tech migrations, logo changes, and schema updates programmatically.
- 🗺️ **Sitemap & Content Discovery:** Map out competitor site hierarchies, product feeds, and RSS updates programmatically.

***

### ❓ Frequently Asked Questions (FAQ)

##### How does Pay-Per-Event billing work?

Under Apify's Pay-Per-Event system, you only pay for actual domains processed and pushed:

- **Standard Extraction:** **$1.50 per 1,000 domains** ($1.00 domain analyzed + $0.50 dataset result pushed, or down to **$1.20 / 1k** on Apify subscription plans).
- **AI-Enriched Extraction:** **$4.50 per 1,000 domains** ($1.00 domain analyzed + $0.50 dataset result pushed + $3.00 AI intelligence, or down to **$3.80 / 1k** on Apify subscription plans).
  There are zero server rental fees or hidden usage costs.

##### Am I charged for domains that fail or cannot be reached?

**No, absolutely not.** If a website returns a network error, DNS failure, or cannot be loaded, it is charged **$0.00**. You only pay for successful results.

##### Is platform usage charged separately?

**No.** Apify compute unit (CU) platform usage is **Free / Included**. You only pay the listed event fee.

##### Can I provide domains without `http://` or `https://`?

**Yes.** You can paste bare domains (`google.com`, `www.nytimes.com`, `sub.domain.org`) or full URLs. The actor automatically normalizes every input into a valid HTTP URL.

##### How does the social profile extractor ensure accuracy?

The actor uses intelligent multi-layer verification to eliminate tracking parameters, embedded sharing widgets, and non-profile links, ensuring you only receive verified company handles and official profiles.

##### Are proxies included?

**Yes.** The actor automatically routes requests through Apify Datacenter Proxies to ensure maximum reliability and bypass anti-bot rate limits.

##### How can I export the results?

Datasets can be exported directly from the Apify Console in **JSON, CSV, Excel (XLSX), XML, or HTML**, or ingested into your CRM or database via the Apify REST API and Webhooks.

***

### 🏷️ Tags & Search Keywords

`website-metadata-extractor` · `ai-industry-classifier` · `seo-analyzer` · `robots-txt-parser` · `sitemap-extractor` · `social-links-scraper` · `jsonld-extractor` · `technology-detector` · `open-graph-scraper` · `twitter-cards` · `meta-tags-scraper` · `lead-enrichment` · `web-intelligence` · `apify` · `crawlee` · `groq`

# Actor input Schema

## `startUrls` (type: `array`):

List of website URLs or domains (e.g. 'apify.com', 'https://shopify.com', 'www.google.com'). Any format is supported.

## `disableDomainAnalysis` (type: `boolean`):

If enabled, the actor will skip external robots.txt and sitemap.xml resolution, analyzing only the on-page HTML for maximum extraction speed.

## `maxConcurrency` (type: `integer`):

Maximum number of domains processed simultaneously. Optimized for high-throughput multi-domain crawling.

## `extractMetaTags` (type: `boolean`):

Extracts title, description, canonical, charset, viewport, OpenGraph, Twitter Cards, theme color, icons, and verification tags.

## `extractRobotsTxt` (type: `boolean`):

Fetches and parses robots.txt into structured per-user-agent allow/disallow rules.

## `extractSitemaps` (type: `boolean`):

Extracts sitemap URLs from robots.txt and verifies the standard /sitemap.xml location.

## `extractSocialLinks` (type: `boolean`):

Performs multi-engine deep scanning across DOM, headers, footers, JSON-LD, and scripts for brand social profiles.

## `extractTechnologies` (type: `boolean`):

Detects CMS (WordPress, Shopify, Wix), Frameworks (React, Next.js, Vue, Nuxt), Analytics, and Web Tools.

## `extractHeadings` (type: `boolean`):

Extracts primary H1, all H1s, all H2s, and all H3 tags for on-page SEO content hierarchy analysis.

## `extractJsonLd` (type: `boolean`):

Parses all Schema.org structured data blocks (Organization, Product, WebSite, FAQ, Article, etc.).

## `extractHreflang` (type: `boolean`):

Extracts international alternate language and regional URL definitions.

## `extractContentMetrics` (type: `boolean`):

Calculates total visible body word count, character count, and language declaration.

## `socialPlatforms` (type: `array`):

Select the social media networks to target for extraction.

## `includeContactInfo` (type: `boolean`):

If enabled, discovers mailto: email addresses and tel: phone links.

## `enableAiClassification` (type: `boolean`):

When enabled, uses high-speed Groq AI (openai/gpt-oss-120b) to extract full company intelligence (industry, summary, B2B/B2C, business model, ICP, core offerings, stage, compliance signals) into the natureOfBusiness field.

## `groqApiKey` (type: `string`):

Optional: Your Groq API key from console.groq.com. If omitted, the Actor will use the GROQ_API_KEY environment variable configured in your Apify Console.

## `customIndustries` (type: `array`):

Optional: Custom list of industries for AI classification. If left empty, the standard list of 100+ global industries is used.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com",
    "www.shopify.com",
    "https://www.wix.com",
    "https://github.com"
  ],
  "disableDomainAnalysis": false,
  "maxConcurrency": 20,
  "extractMetaTags": true,
  "extractRobotsTxt": true,
  "extractSitemaps": true,
  "extractSocialLinks": true,
  "extractTechnologies": true,
  "extractHeadings": true,
  "extractJsonLd": true,
  "extractHreflang": true,
  "extractContentMetrics": true,
  "socialPlatforms": [
    "facebook",
    "twitter",
    "linkedin",
    "instagram",
    "youtube",
    "tiktok",
    "pinterest",
    "reddit",
    "github",
    "discord",
    "telegram",
    "whatsapp",
    "snapchat",
    "threads",
    "medium"
  ],
  "includeContactInfo": false,
  "enableAiClassification": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com",
        "www.shopify.com",
        "https://www.wix.com",
        "https://github.com"
    ],
    "socialPlatforms": [
        "facebook",
        "twitter",
        "linkedin",
        "instagram",
        "youtube",
        "tiktok",
        "pinterest",
        "reddit",
        "github",
        "discord",
        "telegram",
        "whatsapp",
        "snapchat",
        "threads",
        "medium"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("codescraper/website-metadata-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "apify.com",
        "www.shopify.com",
        "https://www.wix.com",
        "https://github.com",
    ],
    "socialPlatforms": [
        "facebook",
        "twitter",
        "linkedin",
        "instagram",
        "youtube",
        "tiktok",
        "pinterest",
        "reddit",
        "github",
        "discord",
        "telegram",
        "whatsapp",
        "snapchat",
        "threads",
        "medium",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("codescraper/website-metadata-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com",
    "www.shopify.com",
    "https://www.wix.com",
    "https://github.com"
  ],
  "socialPlatforms": [
    "facebook",
    "twitter",
    "linkedin",
    "instagram",
    "youtube",
    "tiktok",
    "pinterest",
    "reddit",
    "github",
    "discord",
    "telegram",
    "whatsapp",
    "snapchat",
    "threads",
    "medium"
  ]
}' |
apify call codescraper/website-metadata-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,codescraper/website-metadata-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qPCDnKKRvTRMcppmY/builds/bpaizfrJd5aythrRN/openapi.json