# URL Metadata Extractor: Link Preview & OpenGraph (`thoob/url-metadata-extractor`) Actor

Input URLs, get each page's own public head tags: OpenGraph, Twitter card, title, description, canonical, favicon, and language, for clean link previews and RAG ingestion. Respects robots.txt by default. Billed only per URL successfully read.

- **URL**: https://apify.com/thoob/url-metadata-extractor.md
- **Developed by:** [Pono Data](https://apify.com/thoob) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 4 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 url reads

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL Metadata & OpenGraph Extractor

Give it a list of page URLs and get back the metadata each page publishes for
previews and search: OpenGraph (`og:*`), Twitter card (`twitter:*`), `<title>`,
meta description, meta robots, keywords, author, theme color, published and
modified times, canonical link, declared favicon, page language, hreflang
alternates, feed links, the first h1, and schema.org structured data (JSON-LD
and microdata types). Flat rows, built for link previews and for feeding RAG
pipelines with consistent per-link metadata.

It reads schema.org JSON-LD and microdata types, hreflang alternates and feed
links from the page itself. JSON-LD scripts are parsed as written, and a script
that does not parse is counted in `jsonLdErrors` instead of being guessed at.

### Input

- **URLs**: one per line.
- **Respect robots.txt**: when on (default), the host's robots.txt is checked and
  disallowed URLs are skipped.
- **Max delivered URLs**: cap on billed rows (0 = no cap).

### Output

One row per URL: `url`, `finalUrl`, `httpStatus`, `title`, `description`,
`canonical`, the `og*` fields, the `twitter*` fields, `favicon`, `lang`,
`metaRobots`, `keywords`, `author`, `themeColor`, `publishedTime`,
`modifiedTime`, `hreflang`, `feeds`, `h1`, `h1Count`, `jsonLd`, `jsonLdErrors`,
`jsonLdTypes`, `microdataTypes`, plus provenance (`sourceUrl`, `retrievedAt`,
`confidence`, `dataSource`).

### How it works

Sites publish these tags specifically so other tools can render previews and
read structured data. The actor fetches each page politely with a declared
User-Agent and copies the tags verbatim. The preview fields come from the head
only; JSON-LD, microdata types, and the h1 are also read from the body. Relative
`og:image`, `canonical`, favicon, hreflang, and feed URLs are resolved to
absolute against the page URL; nothing else is transformed, dates are not
reformatted, and a tag the page does not declare is null (or an empty list),
never invented. JSON-LD is capped at 20 objects or 50000 characters per page. A
URL that robots disallows, or that fails to fetch, is written to the free
`rejected` dataset and is not billed. A site owner can ask us to skip their
domain at https://ponodata.com/opt-out ; opted-out hosts are skipped and never
charged.

### Billing

Pay per URL successfully read. A page counts when it has head metadata or at
least one JSON-LD type; an h1 alone does not. Robots-disallowed and failed URLs
cost nothing.

### Sample output

A real run reading each page's own public head tags (one row per URL):

| URL | title | description | OG type |
| --- | --- | --- | --- |
| https://www.cloudflare.com | Cloudflare: Build for the… | Welcome to Cloudflare - Powering … | website |
| https://stripe.com | Stripe / Financial Infras… | Stripe is a financial services pl… | website |
| https://www.python.org | Welcome to Python.org | The official home of the Python P… | website |
| https://kubernetes.io | Kubernetes | Kubernetes, also known as K8s, is… | website |

Every row carries a `sourceUrl` (the page read), for example `https://www.cloudflare.com`. Pages that return no metadata route to the free reject dataset.

### Why this one

- Reads only the page's own public tags and structured data and copies them verbatim, with a `sourceUrl` to the page. Nothing is guessed.
- Follows redirects safely: it records the final URL and screens every redirect hop, so a public URL cannot be turned into a request to an internal host.
- You pay only for a URL that returns metadata. Robots-disallowed and failed URLs are free.
- Typed, flat rows with a stable schema, so an agent or a pipeline can consume them by field name.

### Use cases

- Render link previews at scale: title, description, and og:image for a list of URLs, the same tags a chat app or social card reads.
- Feed a RAG pipeline consistent per-link metadata: one typed row per URL, so a retriever indexes a clean title and description instead of raw HTML.
- Audit a site's social and SEO tags: find pages missing a description, a canonical link, an OpenGraph image, or an h1, or pages with more than one h1 or a noindex robots tag.
- Check schema markup: list the JSON-LD and microdata types each page declares and find ld+json scripts that fail to parse.
- Collect hreflang alternates and RSS, Atom, or JSON Feed links across a list of sites.
- Normalize a mixed link list: each row carries the finalUrl after redirects and the page language.

### FAQ

- Does it read the whole page? It reads the head for the preview and meta fields, and the full document for JSON-LD, microdata types, and the h1. Values are copied verbatim; a tag the page does not declare is null, never invented.
- Does it respect robots.txt? Yes by default; a disallowed URL is skipped to the free `rejected` dataset, unbilled.
- Is it safe against redirect tricks? Yes. It screens every redirect hop, so a public URL that points at an internal address is refused before it connects.
- How am I billed? Per URL that returns metadata or JSON-LD types. Robots-disallowed and failed URLs cost nothing.

### See also

More pay-only-for-results data tools from Pono Data:

- [Sitemap Extractor](https://apify.com/thoob/sitemap-extractor) - every URL from any sitemap
- [Bulk DNS Lookup](https://apify.com/thoob/dns-bulk-lookup) - DNS records plus SPF, DMARC, and CAA
- [Domain WHOIS via RDAP](https://apify.com/thoob/rdap-domain-lookup) - registration data, structured from RDAP

Full catalog: https://apify.com/thoob

# Actor input Schema

## `urls` (type: `array`):

Page URLs to read metadata from, one per line. The actor fetches each page's HTML head and extracts the tags the site publishes for previews.

## `respectRobots` (type: `boolean`):

Check the host's robots.txt before fetching and skip any URL it disallows for our agent. Recommended on.

## `maxUrls` (type: `integer`):

Cap on delivered, billed rows. 0 means no cap. The platform spend cap is honored regardless.

## Actor input object example

```json
{
  "urls": [
    "https://github.com",
    "https://www.bbc.com/news"
  ],
  "respectRobots": true,
  "maxUrls": 0
}
```

# Actor output Schema

## `pages` (type: `string`):

One row per URL read.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://github.com",
        "https://www.bbc.com/news"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("thoob/url-metadata-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://github.com",
        "https://www.bbc.com/news",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("thoob/url-metadata-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://github.com",
    "https://www.bbc.com/news"
  ]
}' |
apify call thoob/url-metadata-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thoob/url-metadata-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pH9MBObsNIX8TlIfR/builds/O2sywShBBAXbwXHWn/openapi.json