# Article Extractor: Paywall, Split Pages & Tables (`aiqlabs/article-extractor`) Actor

Extracts article text and Markdown, and reports three things other extractors stay quiet about: a paywall that cut the body short, an article that continues on later pages, and tables and links that plain-text output destroys.

- **URL**: https://apify.com/aiqlabs/article-extractor.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 4 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Article Extractor: Paywall, Split Pages & Tables

Give it article URLs. It returns the body as Markdown and plain text, with the
metadata — and it tells you three things the other extractors do not.

### What is different here

Every extractor on this store returns "clean article text". So does this one. The
difference is what it says about that text.

#### 1. Whether a paywall cut the body short

An extractor that returns the first three paragraphs of a metered article and calls
it a success is worse than one that fails, because the failure is invisible. Feed a
thousand of those into a corpus and it looks complete.

This Actor reports two separate fields, because they come apart constantly:

| Field | Meaning |
|---|---|
| `paywallDetected` | The page runs a paywall |
| `bodyTruncated` | **The text you are holding is short of what the page itself claims** |

Publishers who want to stay in Google ship the whole article in the HTML and hide it
with CSS, so a page can be paywalled with the body complete. A metered site on your
last free view ships three paragraphs with no paywall markup at all. Only the second
field is the one that should stop your pipeline.

Measured on a live New York Times article: `paywallDetected: true` from three
signals (`isAccessibleForFree: false`, `hasPart` naming `.meteredContent`, and a
`meteredcontent` class in the markup), `bodyTruncated: false`, 2,757 words extracted.

**It does not bypass anything.** The page is fetched the way any reader's browser
fetches it, and the only output is a verdict on what arrived. There is no login, no
cookie injection and no cache trick.

Truncation is proven against the page's own `articleBody` or `wordCount` in JSON-LD
when it publishes one — `coverageRatio` and `coverageBasis` show the arithmetic — and
otherwise inferred from named signals you can read in `paywallEvidence`.

#### 2. Whether the article continues on later pages

Long articles get split, and `rel="next"` is used for two unrelated things: the rest
of this article, and the next page of an index. Following it blindly appends a
different article to the body.

So a continuation has to earn the merge:

- same site, never a cross-domain hop
- both pages claim the same title, after `Page 2 of 5` decoration is removed
- the source page is not itself an index

When a merge is declined the reason is in `issues`, not swallowed. Real refusals from
the test run:

```
not the same article: titles differ ("html 4 01 specification" vs "about the html 4 specification")
this page is an index, so its "next" link is site pagination rather than a continuation
the link holds unrendered template syntax (/'.$next.')
```

That last one is a live WordPress blog whose `rel="next"` is literally
`'.$next.'` — valid as a relative URL, so nothing but a look at the string catches it.

When parts are merged, the header and footer each part repeats are dropped
(`repeatedBlocksRemoved`), and only at the edges of a part — a sentence repeated in
the middle of a body is the author, not furniture.

#### 3. Tables, links and code, kept

"Clean text" is where a table goes to die. A 40-row financial table becomes a run of
loose numbers with no columns, and the article's whole point goes with it.

Markdown output keeps:

- **tables** as Markdown tables, with the header row taken from the document's own
  `<thead>` or `<th>` rather than guessed, and `colspan` widened so values stay under
  their own column
- **links** as `[text](absolute-url)`, resolved against the page
- **code blocks** with their line breaks and language, from the `<pre>` as written
- nested lists, block quotes, figure captions, definition lists

`tableCount`, `linkCount` and `codeBlockCount` let you tell "this article has three
tables" from "this article had three tables".

#### 4. And: published or last updated

Everyone returns "publish date". A page rewritten two years after publication looks
identical in that output. This Actor returns `publishedAt` and `modifiedAt`
separately, plus `updatedAfterPublish` and `daysBetweenPublishAndUpdate`. Stamps
written minutes apart are the CMS saving both on publish, not a revision, so they do
not count.

### Also included

- **Is it even an article?** `pageKind` separates `article` from `listing`. A category
  page with eight teasers is not an article; a blog post with a "you may also like"
  rail is, even though both trip the same signals.
- **`extractionConfidence`** (`high` / `medium` / `low`) with `extractedVia` naming
  what chose the body: an `<article>` element, `itemprop="articleBody"`, or paragraph
  scoring.
- Title taken by **vote** across JSON-LD `headline`, `og:title`, `<title>` and `<h1>`
  rather than by rank. Wikipedia puts the Wikidata one-line description in `headline`,
  so ranking it first titles the web scraping article "data scraping used for
  extracting data from websites".
- Author, site name, section, language, description, hero image, tags, canonical,
  schema.org types, word count, reading time.
- **CJK counted per character**, so a Japanese article is not measured as one word.
- **robots.txt is honoured.** A disallowed page is reported with the rule that
  blocked it rather than dropped silently — and not downloaded.

### Input

```jsonc
{
  "urls": ["https://example.com/post"],
  "followSplitPages": true,        // merge continuation pages
  "maxPagesPerArticle": 10,
  "outputFormats": ["markdown", "text"],   // also "html" for the cleaned article HTML
  "respectRobotsTxt": true,
  "skipTruncatedByPaywall": false, // drop partial bodies instead of returning them
  "maxConcurrency": 8
}
```

Set `skipTruncatedByPaywall` when the output feeds a corpus that must not contain
half articles. Off by default so the finding stays visible.

### Output

One row per input URL. `SUMMARY` in the key-value store holds the totals: how many
were paywalled, how many came back cut short, how many were merged from several
pages, how many were not articles.

### Cost

**Pay per event: $0.00005 per run plus $0.005 per article returned** (in effect since 4 August 2026). One request per article, plus one
robots.txt per host, plus one request per continuation page actually merged. No
browser, no JavaScript execution, no proxy.

### Verification

- 69 unit tests
- 26 checks against the live web on every release, covering a blog post, a
  reference page heavy with tables and links, a paywalled news article, a section
  page, a blog index, a Japanese WordPress post and a chaptered specification —
  including the negative controls that an index is not merged as one article and a
  free post is not reported as paywalled

Defects these live checks found and that the unit tests did not: a blog index whose
six pagination pages were merged into "one article", a WordPress post reported as
`not_an_article` while holding a full body, Wikipedia titled by its JSON-LD
description, MediaWiki maintenance notices treated as body text, and a crash in
Node's own HTTP client when a keep-alive socket closes (`assert(!this.paused)` in
undici's HTTP/1 parser) — measured across eight consecutive runs per approach and
guarded explicitly, since it fires from a socket event where nothing can catch it.

# Actor input Schema

## `urls` (type: `array`):

Article, blog post or news pages to extract. Full URLs (https://example.com/post) or bare hosts both work, and one entry may hold several URLs separated by newlines, commas or spaces.

## `followSplitPages` (type: `boolean`):

When a page declares a continuation (rel="next", or a "Page 2 of 5" style pager), fetch the rest and return one merged body. The repeated header and footer of each part are removed before merging.

## `maxPagesPerArticle` (type: `integer`):

Upper bound on how many continuation pages one article may pull in, so a mislabelled pager cannot walk a whole site.

## `outputFormats` (type: `array`):

Which representations of the body to include. Markdown keeps tables, links, lists, quotes and code; text is plain reading order; html is the cleaned article HTML.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that robots.txt disallows. The URL is still reported, with the rule that blocked it, instead of being dropped silently.

## `skipTruncatedByPaywall` (type: `boolean`):

Leave partial articles out of the dataset instead of returning them. Off by default so the finding stays visible; turn it on when the output feeds a corpus that must not contain half articles.

## `maxConcurrency` (type: `integer`):

How many articles to fetch at the same time. Lower this for small servers.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one response before giving up on it.

## `maxUrls` (type: `integer`):

Hard cap on how many input URLs are processed in one run, counted after duplicates are removed.

## `maxHtmlKbPerPage` (type: `integer`):

How much of each page's HTML to read. Raise it if a row reports "html_read_limit_reached" — that means the end of the document, and anything in it, was cut off. Lower it to cap memory on very large crawls.

## Actor input object example

```json
{
  "urls": [
    "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "followSplitPages": true,
  "maxPagesPerArticle": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "respectRobotsTxt": true,
  "skipTruncatedByPaywall": false,
  "maxConcurrency": 8,
  "requestTimeoutSecs": 20,
  "maxUrls": 5000,
  "maxHtmlKbPerPage": 6144
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/article-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/article-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ]
}' |
apify call aiqlabs/article-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aiqlabs/article-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/am63LbH3KYqNEy6Cf/builds/3rLq8l3z679Kf7Xp6/openapi.json