# Web Content Extractor API — URL to JSON (`george.the.developer/web-content-extractor-api`) Actor

Extract structured JSON from any webpage. Articles, products, recipes, jobs. Auto-detects content type. Returns metadata, headings, images, links. For AI agents and RAG.

- **URL**: https://apify.com/george.the.developer/web-content-extractor-api.md
- **Developed by:** [George Kioko](https://apify.com/george.the.developer) (community)
- **Categories:** Lead generation, Marketing
- **Features:** Limited permissions, Single-tenant, Server interface
- **Stats:** 21 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.40 / 1,000 content extractions

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🔍 Web Content Extractor API — URL to Structured JSON

> **One API call. Any URL. Clean structured JSON.** Extract articles, products, recipes, job postings, and more — automatically detected and organized. Built for AI agents, RAG pipelines, and data workflows.

***

### Architecture Overview

```mermaid
flowchart TB
    subgraph Input
        URL[/"URL: any webpage"/]
    end

    subgraph Processing["Extraction Pipeline"]
        FETCH["1. Fetch & Parse HTML"]
        DETECT["2. Auto-Detect Content Type"]
        SCORE["3. Score Content Blocks"]
        EXTRACT["4. Extract Structured Data"]
        ENRICH["5. Enrich with Metadata"]
    end

    subgraph Detection["Content Type Detection"]
        ART["Article"]
        PROD["Product"]
        REC["Recipe"]
        JOB["Job Posting"]
        EVT["Event"]
        WEB["Generic Webpage"]
    end

    subgraph Output["Structured JSON"]
        META["Metadata: title, author, date, image"]
        CONTENT["Content: text, headings, word count"]
        MEDIA["Media: images, links"]
        SCHEMA["JSON-LD Structured Data"]
        TYPED["Type-Specific: price, ingredients, salary..."]
    end

    URL --> FETCH --> DETECT --> SCORE --> EXTRACT --> ENRICH
    DETECT --> ART & PROD & REC & JOB & EVT & WEB
    ENRICH --> META & CONTENT & MEDIA & SCHEMA & TYPED

    style Input fill:#1a1a2e,color:#fff
    style Processing fill:#16213e,color:#fff
    style Detection fill:#0f3460,color:#fff
    style Output fill:#533483,color:#fff
```

### What Makes This Different?

| Feature | This Actor | Typical Scrapers |
|---------|-----------|-----------------|
| Output format | **Structured JSON** | Raw HTML |
| Content detection | **Auto-detects 6 types** | Manual configuration |
| Setup time | **Zero** — just pass URL | Hours of selector writing |
| AI-ready | **Yes** — clean text for LLMs | Needs post-processing |
| Batch support | **Up to 25 URLs** per call | One at a time |
| Response time | **1-3 seconds** | 5-30 seconds |

***

### Request Flow

```mermaid
sequenceDiagram
    participant Client as Your App
    participant API as Content Extractor
    participant Web as Target Website
    participant Cache as 30-min Cache

    Client->>API: GET /extract?url=example.com
    API->>Cache: Check cache

    alt Cache Hit
        Cache-->>API: Return cached result
        API-->>Client: JSON response (instant)
    else Cache Miss
        API->>Web: Fetch HTML
        Web-->>API: HTML content
        API->>API: Detect type + Extract + Score
        API->>Cache: Store result
        API-->>Client: Structured JSON (1-3s)
    end

    Note over Client,API: PPE charge: $0.003 per extraction
```

***

### API Endpoints

#### `GET /extract` — Extract from URL

```
GET /extract?url=https://techcrunch.com/2026/03/24/ai-news&format=full
```

| Parameter | Type | Required | Default | Options |
|-----------|------|----------|---------|---------|
| `url` | string | Yes | — | Any valid URL |
| `format` | string | No | `full` | `full`, `article`, `metadata` |

#### `POST /extract` — Extract with JSON body

```json
POST /extract
{
  "url": "https://techcrunch.com/2026/03/24/ai-news",
  "format": "article"
}
```

#### `POST /batch` — Extract multiple URLs

```json
POST /batch
{
  "urls": [
    "https://news.ycombinator.com",
    "https://techcrunch.com",
    "https://bbc.com/news"
  ],
  "format": "full"
}
```

#### `GET /` — Health check

Returns API status, version, and endpoint documentation.

***

### Content Type Detection

```mermaid
flowchart LR
    HTML["HTML Page"] --> CHECK{"Detect Signals"}

    CHECK -->|"og:type=article<br/>or article tag"| ART["**article**<br/>title, author, date,<br/>full text, headings"]
    CHECK -->|"Schema: Product<br/>or .product-price"| PROD["**product**<br/>name, price, rating,<br/>images, SKU, brand"]
    CHECK -->|"Schema: Recipe<br/>or .recipe"| REC["**recipe**<br/>ingredients, instructions,<br/>prep time, servings"]
    CHECK -->|"Schema: JobPosting<br/>or .job-title"| JOB["**job_posting**<br/>title, company, salary,<br/>location, type"]
    CHECK -->|"Schema: Event<br/>or .event-date"| EVT["**event**<br/>name, date, location,<br/>description"]
    CHECK -->|"No specific<br/>signals found"| WEB["**webpage**<br/>metadata, content,<br/>links, images"]

    style ART fill:#10b981,color:#fff
    style PROD fill:#f59e0b,color:#fff
    style REC fill:#ef4444,color:#fff
    style JOB fill:#3b82f6,color:#fff
    style EVT fill:#8b5cf6,color:#fff
    style WEB fill:#6b7280,color:#fff
```

***

### Output Examples

#### Article Extraction

```json
{
  "url": "https://techcrunch.com/2026/03/24/ai-agents",
  "type": "article",
  "metadata": {
    "title": "AI Agents Are Reshaping Enterprise Software",
    "description": "How autonomous AI agents are changing B2B SaaS",
    "author": "Sarah Perez",
    "date": "2026-03-24T10:00:00Z",
    "image": "https://techcrunch.com/hero.jpg",
    "siteName": "TechCrunch",
    "locale": "en-US",
    "canonical": "https://techcrunch.com/2026/03/24/ai-agents",
    "keywords": ["AI", "agents", "enterprise", "SaaS"]
  },
  "content": {
    "text": "The rise of AI agents represents a fundamental shift in how enterprise software operates. Unlike traditional chatbots...",
    "headings": [
      { "level": 2, "text": "What Are AI Agents?" },
      { "level": 2, "text": "The Enterprise Impact" },
      { "level": 3, "text": "Case Study: Salesforce" }
    ],
    "wordCount": 2847
  },
  "media": {
    "images": [
      { "src": "https://techcrunch.com/diagram.png", "alt": "AI agent architecture" }
    ],
    "links": [
      { "href": "https://openai.com/agents", "text": "OpenAI's agent framework" }
    ]
  },
  "structuredData": [{ "@type": "NewsArticle", "headline": "..." }],
  "extractedAt": "2026-03-24T12:34:56.789Z"
}
```

#### Product Extraction

```json
{
  "url": "https://store.example.com/product/widget-pro",
  "type": "product",
  "metadata": { "title": "Widget Pro - Best Seller", "siteName": "Example Store" },
  "content": { "text": "The Widget Pro is our most popular...", "wordCount": 342 },
  "product": {
    "name": "Widget Pro",
    "price": "$49.99",
    "currency": "USD",
    "availability": "InStock",
    "rating": "4.8",
    "reviewCount": "1,247",
    "brand": "WidgetCo",
    "sku": "WP-2026",
    "images": ["https://store.example.com/widget-pro-1.jpg"]
  }
}
```

***

### Use Case Workflows

#### RAG Pipeline Integration

```mermaid
flowchart LR
    URLs["URL List<br/>100+ sources"] --> EXTRACT["Web Content<br/>Extractor API"]
    EXTRACT --> TEXT["Clean Text<br/>+ Metadata"]
    TEXT --> CHUNK["Text Chunking<br/>(LangChain)"]
    CHUNK --> EMBED["Embeddings<br/>(OpenAI)"]
    EMBED --> VECTOR["Vector DB<br/>(Pinecone)"]
    VECTOR --> RAG["RAG Query<br/>Engine"]
    RAG --> ANSWER["AI-Powered<br/>Answers"]

    style EXTRACT fill:#10b981,color:#fff
    style RAG fill:#3b82f6,color:#fff
```

#### Competitive Intelligence Pipeline

```mermaid
flowchart LR
    COMP["Competitor<br/>URLs"] --> EXTRACT["Web Content<br/>Extractor API"]
    EXTRACT --> PROD["Product Data:<br/>prices, features"]
    EXTRACT --> NEWS["News & Blog:<br/>announcements"]
    PROD --> DASH["Analytics<br/>Dashboard"]
    NEWS --> ALERT["Email<br/>Alerts"]

    style EXTRACT fill:#10b981,color:#fff
```

***

### Pricing

| Event | Price per call | Cost per 1,000 |
|-------|---------------|-----------------|
| **Content extraction** | $0.003 | $3.00 |

#### Cost Comparison

| Solution | Cost per 1,000 URLs | Setup Time |
|----------|-------------------|------------|
| **This Actor** | **$3.00** | **0 minutes** |
| Diffbot | $299/month flat | Hours |
| Custom scraper | $50+ developer hours | Days |
| Manual copy-paste | 40+ hours labor | Forever |

***

### Integrations

| Platform | How to Connect |
|----------|---------------|
| **LangChain** | Use as Document Loader via HTTP |
| **LlamaIndex** | Custom reader pointing to /extract |
| **Zapier** | Webhook trigger -> GET /extract |
| **Make (Integromat)** | HTTP module -> POST /extract |
| **n8n** | HTTP Request node |
| **Apify Orchestrator** | Direct actor call or Standby URL |

***

### FAQ

**Q: How fast is extraction?**
A: 1-3 seconds for a single URL. Batch processes 25 URLs in parallel.

**Q: Does it handle paywalled content?**
A: It extracts whatever is publicly visible in the HTML. Paywalled content behind JavaScript auth won't be extracted.

**Q: What about JavaScript-rendered pages (SPAs)?**
A: Current version uses server-side HTML. For JS-heavy pages, pair with our [Screenshot & PDF API](https://apify.com/george.the.developer/screenshot-pdf-api).

**Q: Is there a rate limit?**
A: No hard rate limit. Apify Standby handles concurrent requests automatically.

**Q: What languages are supported?**
A: Any language. The extractor works with HTML structure, not language-specific parsing.

***

### Related Actors

- [WebSight API](https://apify.com/george.the.developer/websight-api) — Technical website analysis (SEO, tech stack, AI score)
- [Screenshot & PDF API](https://apify.com/george.the.developer/screenshot-pdf-api) — Pixel-perfect webpage captures
- [Website Contact Scraper](https://apify.com/george.the.developer/website-contact-scraper) — Extract emails, phones, social links

***

*Built by [George Kioko](https://apify.com/george.the.developer) | 6,196+ data extraction jobs completed | 35+ production APIs*

# Actor input Schema

## `url` (type: `string`):

URL to extract content from

## `format` (type: `string`):

Output format: 'full' (all fields), 'article' (text-focused), 'metadata' (meta only)

## Actor input object example

```json
{
  "url": "https://example.com",
  "format": "full"
}
```

# Actor output Schema

## `results` (type: `string`):

Rows of this run's dataset in the Overview table view, with url, title, type, author, date and words.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://example.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("george.the.developer/web-content-extractor-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "url": "https://example.com" }

# Run the Actor and wait for it to finish
run = client.actor("george.the.developer/web-content-extractor-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://example.com"
}' |
apify call george.the.developer/web-content-extractor-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,george.the.developer/web-content-extractor-api"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kEMqxf4ZWVeRGh7gK/builds/HxQmVIS9AbE6E7FRc/openapi.json