# web-content-extractor (`morph_coder/web-content-extractor`) Actor

HTTP-first web scraper without proxy by default. Returns clean article text, SEO metadata, and headings. Apify Proxy and Playwright used only as fallback. No-code friendly JSON output for Make, n8n, and Zapier

- **URL**: https://apify.com/morph_coder/web-content-extractor.md
- **Developed by:** [Morph Coder](https://apify.com/morph_coder) (community)
- **Categories:** Automation, SEO tools, Lead generation
- **Features:** Limited permissions
- **Stats:** 34 total users, 5 monthly users, 99.8% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $2.00 / 1,000 record scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Content Extractor

Apify Actor for **no-code friendly** web content extraction. Fetches pages **without proxy by default** (cheap HTTP), uses Apify Proxy and Playwright **only as fallback**.

### Features

- **HTTP-first** — direct Cheerio fetch without proxy (no proxy cost on most pages)
- **4-step fallback** — Cheerio → Cheerio+Proxy → Playwright → Playwright+Proxy
- **Clean content** — Mozilla Readability removes nav, banners, and noise
- **SEO metadata** — title, meta description, OG tags, canonical, all headings (h1–h6)
- **No-code output** — flat JSON fields for Make, n8n, Zapier
- **Social URL blocking** — Facebook, YouTube, etc. return `blocked_social` without scraping

### Input

| Field | Default | Description |
|-------|---------|-------------|
| `urls` | — | **Required.** List of page URLs |
| `outputFormat` | `json` | `json` | `text` | `markdown` |
| `useProxyFallback` | `true` | Retry with Apify Proxy on HTTP errors / blocks |
| `usePlaywrightFallback` | `true` | Retry thin/JS pages with headless Chrome |
| `minContentLength` | `200` | Min text length before Playwright fallback |
| `country` | `US` | Proxy country (fallback steps only) |
| `proxyType` | `RESIDENTIAL` | `RESIDENTIAL` or `DATACENTER` |
| `batchSize` | `50` | URLs per processing wave |
| `maxUrls` | `1000` | Hard cap per run |
| `extractExternalLinks` | `false` | One URL per external domain |

#### Example input

```json
{
  "urls": [
    "https://example.com/article",
    "https://blog.example.org/post"
  ],
  "outputFormat": "json"
}
```

### Output

One dataset item per URL.

#### Success example (`outputFormat: json`)

```json
{
  "url": "https://example.com/article",
  "success": true,
  "status": "success_static",
  "method": "cheerio",
  "usedProxy": false,
  "title": "Article title",
  "metaDescription": "SEO description from meta tag",
  "ogTitle": "Article title",
  "ogDescription": "SEO description",
  "canonical": "https://example.com/article",
  "headings": [
    { "level": 1, "text": "Main heading" },
    { "level": 2, "text": "Section" }
  ],
  "fullText": "Clean plain text of the article...",
  "content": "Clean plain text of the article...",
  "markdown": "# Main heading\n\nClean plain text...",
  "wordCount": 842,
  "httpStatus": 200,
  "fetchedAt": "2026-06-24T12:00:00.000Z",
  "billable": true
}
```

#### `outputFormat` behavior

| Format | `content` field | Other fields |
|--------|-----------------|--------------|
| `json` | Plain text (`fullText`) | All SEO fields + `markdown` |
| `text` | Plain text | `title`, `success` only (SEO fields null) |
| `markdown` | Markdown | All SEO fields |

#### Status values

| Status | Meaning |
|--------|---------|
| `success_static` | Cheerio HTTP fetch succeeded |
| `success_rendered` | Playwright rendering succeeded |
| `blocked_social` | Social URL — not scraped |
| `failed_fetch` | HTTP or network error |
| `failed_dynamic` | Content too short / not extractable |
| `failed_timeout` | Request timed out |

### Fallback chain

```
URLs → Cheerio (no proxy)
     → [403/block/timeout] → Cheerio (Apify Proxy)
     → [thin content]      → Playwright (no proxy)
     → [still failing]     → Playwright (Apify Proxy)
```

Proxy is initialized **lazily** — only when a fallback step needs it.

### No-code integration

#### Make (Integromat)

1. Add **Apify** module → **Run Actor** → `web-content-extractor`
2. Pass `urls` array
3. Map output fields directly:
   - `content` → text field in Google Docs / Notion
   - `metaDescription` → SEO field
   - `headings` → iterator for outline generation
4. Filter: `success` = `true`

#### n8n

1. **Apify** node → Run Actor
2. Use **JSON** output — no markdown parsing needed
3. Expression: `{{ $json.content }}`, `{{ $json.metaDescription }}`
4. **IF** node: `{{ $json.success }}` equals `true`

#### Zapier

1. **Apify** action → Run Actor
2. Filter Zap: only continue if `success` is `true`
3. Map `content` and `title` to next step (e.g. OpenAI, Airtable)

### Local development

```bash
npm install
npm run build
npm run dev
```

Set input via Apify CLI or `storage/key_value_stores/default/INPUT.json`:

```json
{
  "urls": ["https://example.com"]
}
```

### Docker

Built on `apify/actor-node-playwright-chrome` for Playwright fallback support.

```bash
apify push
```

### Billing

Pay-per-event: one `scraped-url` charge per successfully extracted URL. Proxy and Playwright fallback steps do not add extra charges.

# Actor input Schema

## `urls` (type: `array`):

List of direct page URLs. Social network URLs are returned as blocked_social without scraping.

## `outputFormat` (type: `string`):

json: plain text content + SEO fields (best for Make/n8n). text: minimal fields. markdown: markdown content + SEO fields.

## `useProxyFallback` (type: `boolean`):

When the direct HTTP request fails (403, block page, timeout), retry with Apify Proxy before Playwright.

## `country` (type: `string`):

Two-letter country code for Apify proxy geolocation — used only on proxy fallback steps (e.g. US, DE, UA).

## `proxyType` (type: `string`):

Apify proxy group — used only on proxy fallback steps.

## `usePlaywrightFallback` (type: `boolean`):

Retry thin or JS-rendered pages with headless Chrome.

## `minContentLength` (type: `integer`):

Extracted text shorter than this triggers Playwright fallback (if enabled).

## `batchSize` (type: `integer`):

Number of URLs processed per wave (memory control).

## `maxUrls` (type: `integer`):

Hard cap on URLs accepted per run.

## `extractExternalLinks` (type: `boolean`):

One sample URL per external domain, filtered for brand/leak checks. Off by default.

## `maxExternalLinks` (type: `integer`):

Cap on unique outbound domains when extractExternalLinks is true.

## `blockHeavyResources` (type: `boolean`):

When Playwright fallback runs, block images and web fonts via Firefox prefs to cut proxy bandwidth. Scripts, XHR, CSS, and documents still load (needed for antibot). Does not use page.route (which can trip WAFs on Camoufox). Turn off if a site fails only with this enabled.

## Actor input object example

```json
{
  "urls": [
    "https://example.com"
  ],
  "outputFormat": "json",
  "useProxyFallback": true,
  "country": "US",
  "proxyType": "RESIDENTIAL",
  "usePlaywrightFallback": true,
  "minContentLength": 200,
  "batchSize": 50,
  "maxUrls": 1000,
  "extractExternalLinks": false,
  "maxExternalLinks": 100,
  "blockHeavyResources": true
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset with content and SEO fields per URL.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com"
    ],
    "country": "US"
};

// Run the Actor and wait for it to finish
const run = await client.actor("morph_coder/web-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://example.com"],
    "country": "US",
}

# Run the Actor and wait for it to finish
run = client.actor("morph_coder/web-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com"
  ],
  "country": "US"
}' |
apify call morph_coder/web-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,morph_coder/web-content-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jxgfBXzdHYl0xS9Ss/builds/JSnvbGNUmT4cQSOwm/openapi.json