Article Content Extractor — Clean Text, Markdown & Metadata

Article Content Extractor — Clean Text, Markdown & Metadata for AI & RAG

Transform any article, blog post, documentation page, or news URL into sanitized, highly structured content. This article content extractor and news scraper allows you to input a single URL or thousands of links and extract the core article body — headline, author byline, publication timestamp, clean plain text, LLM-ready Markdown, lead image, language code, word count, and reading duration — with navigation menus, banners, sponsored widgets, author boxes, cookie prompts, and clutter systematically stripped away. Zero browser overhead, no login requirements, no custom scrapers.

Process articles in bulk from leading newsrooms, technical publications, enterprise blogs, and encyclopedias, and export clean datasets directly to JSON, CSV, Excel, XML, or via REST API.

---

⚡ At a glance

Feature	Details & Support
Primary Function	URL-to-Article extraction, boilerplate elimination, document cleanup
Input Sources	Direct URL lists, CSV/text file uploads, Google Sheets, Apify Request Lists
Output Formats	Clean Markdown (.md), Plain Text, Cleaned HTML, Structured JSON metadata
Target Sites	News publishers, tech blogs, documentation, academic portals, Substack, Medium
Batch Capacity	High-throughput parallel ingestion unlocked for 1,000+ URLs per run
Metadata Parsing	JSON-LD, OpenGraph, microdata, HTML header fallback extraction
AI / RAG Ready	Formatted Markdown headings, bullet lists, fenced code blocks, and absolute link resolution
Automation	Apify Cron, Webhooks, Make.com, Zapier, LangChain Document Loaders, LlamaIndex

---

🔎 Why scrape articles with Article Content Extractor?

Modern web pages are flooded with non-content elements: sticky navigation bars, header ads, interactive popups, newsletter signups, GDPR cookie banners, sponsored content carousels, and footer links. When feeding raw HTML into Large Language Models (LLMs) or vector databases, this extraneous "page furniture" burns context window tokens, dilutes semantic search embeddings, and introduces severe hallucinations.

Extracting the essential article content — the authentic headline, author byline, publication timestamp, and narrative text — typically requires maintaining brittle per-site CSS selectors that break with every layout redesign.

This article content extractor resolves this problem permanently. You provide the target URLs, and the actor executes intelligent readability analysis:

• 🤖 LLM & RAG Ingestion: Deliver pristine Markdown chunks directly to embedding models (OpenAI, Cohere, Voyage) without token bloat.
• 🧠 AI Knowledge Base Construction: Continuously refresh corporate knowledge repositories from verified web publications and industry whitepapers.
• 📰 Competitive News & PR Monitoring: Harvest full text from hundreds of competitor press releases, industry blogs, and editorial articles for sentiment analysis.
• 📚 Academic & Research Citations: Archive verifiable text accompanied by definitive author, publishedDate, and siteName provenance.
• 📬 Automated Editorial Newsletters: Aggregate verified excerpts, titles, and high-resolution lead images to curate automated digest feeds.
• 🔍 SEO & Content Architecture Audits: Quantify word counts, reading speeds, headline hierarchies, and meta structures across thousands of digital assets.

---

⚖️ Article Content Extractor vs. Generic Crawlers

Feature	Article Content Extractor	Generic Web Crawlers
LLM-Ready Markdown	✅ Clean Markdown with preserved headers & code blocks	❌ Raw messy text or raw tag soup
Boilerplate Stripping	✅ Automatic removal of ads, banners, footers & menus	❌ Retains navigation, cookie notices, and ads
Resource Efficiency	✅ Lightweight HTTP engine (sub-second per article)	❌ Heavy browser instances consuming gigabytes of RAM
Metadata Quality	✅ Layered JSON-LD, OpenGraph, and microdata synthesis	❌ Basic title and meta description only
Relative URL Fixes	✅ Automatically converts relative image/link paths to absolute	❌ Broken relative links (/images/...)
Predictable Scope	✅ URL-driven: extracts only the exact URLs you provide	❌ Risks runaway crawls across unintended subdomains

---

✨ Features & Extraction Capabilities

• 📄 Pure Main-Content Extraction: Isolates the primary article body while purging extraneous navigation, footers, advertisements, and sidebar links.
• 📝 Markdown Optimized for LLMs: Translates article layouts into clean ATX Markdown (# Heading, - list, code), ideal for vectorization.
• 🧹 Normalized Plain Text: Delivers whitespace-sanitized text ready for traditional NLP tokenization, classification, and entity extraction.
• 🏷️ Layered Metadata Extraction: Combines JSON-LD (Article, NewsArticle, BlogPosting), OpenGraph tags, and HTML attributes to capture authors, publication dates, and keywords.
• 🖼️ Absolute Media Resolution: Normalizes all image sources and hyperlinks within the content body to absolute, resolvable URLs.
• ⏱️ Built-in Quality Metrics: Automatically calculates wordCount and readingTimeMinutes to facilitate automated quality filtering.
• ⚡ High Concurrency & Resilience: Configurable parallel workers (1–20 concurrent requests) with automatic retries and proxy session rotation.

---

🎯 High-Impact Use Cases

1. Retrieval-Augmented Generation (RAG) Pipelines

Extract documentation, tech tutorials, and market reports as structured Markdown. Eliminate token waste caused by cookie prompts and navigation bars, maximizing prompt context efficiency.

2. Media Monitoring & Corporate Intelligence

Feed thousands of media links into automated sentiment classification and entity recognition pipelines without maintaining scraping rules for each media outlet.

3. Content Archiving & Research Libraries

Create immutable, clutter-free text archives of academic essays, investigative journalism, and industry reports with verified author and timestamp metadata.

4. News Aggregation & Curated Feeds

Extract clean headlines, 200-word summaries, and primary feature graphics to populate niche community feeds and automated weekly briefing newsletters.

---

📥 Input Parameters

The actor accepts clean JSON configurations via the Apify Console or API:

Parameter	Type	Default	Description
startUrls	Array	[...]	Article, blog, or news URLs to extract. Supports direct arrays, CSV uploads, or Google Sheets.
includeMarkdown	Boolean	true	When true, returns clean, structured Markdown representation of the article body.
includeText	Boolean	true	When true, provides sanitized plain-text version of the article body.
includeHtml	Boolean	false	When true, includes cleaned article HTML without surrounding page boilerplate.
includeImages	Boolean	true	When true, gathers all image URLs embedded within the article body.
maxItems	Integer	100	Maximum number of articles to process in this run (0 = unlimited, processes all URLs).
maxConcurrency	Integer	8	Parallel request concurrency (recommended 6–12).
proxyConfiguration	Object	{ "useApifyProxy": true }	Apify Proxy configuration for high-availability extraction across diverse publishers.

Input JSON Example (RAG Markdown Extraction)

{
  "startUrls": [
    { "url": "https://blog.apify.com/what-is-web-scraping/" },
    { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }
  ],
  "includeMarkdown": true,
  "includeText": false,
  "includeHtml": false,
  "includeImages": true,
  "maxItems": 50,
  "maxConcurrency": 8,
  "proxyConfiguration": { "useApifyProxy": true }
}

---

📤 Output Data Structure

Each processed URL generates exactly one record in the default Apify dataset.

Sample Output Record

{
  "url": "https://blog.apify.com/what-is-web-scraping/",
  "finalUrl": "https://blog.apify.com/what-is-web-scraping/",
  "title": "What is Web Scraping and How Does It Work?",
  "author": "Apify Editorial Team",
  "publishedDate": "2026-03-15T08:00:00.000Z",
  "modifiedDate": "2026-10-20T14:30:00.000Z",
  "excerpt": "A complete beginner's guide to web scraping, data extraction, legal considerations, and modern tooling in 2026.",
  "siteName": "Apify Blog",
  "lang": "en",
  "keywords": "web scraping, data extraction, apify, crawlee, python, javascript",
  "mainImage": "https://blog.apify.com/content/images/web-scraping-guide.png",
  "wordCount": 1845,
  "readingTimeMinutes": 8,
  "text": "Web scraping is the process of automatically extracting data and structured information from websites...",
  "markdown": "## What is Web Scraping?\n\nWeb scraping is the process of **automatically extracting data** from websites...\n\n### Core Applications\n\n- Lead generation\n- Market intelligence\n- AI training data",
  "images": [
    "https://blog.apify.com/content/images/scraping-architecture.png",
    "https://blog.apify.com/content/images/crawler-diagram.png"
  ],
  "extractedAt": "2026-11-10T13:20:00.000Z"
}

📋 Output Fields Specification & Coverage

Field Category	Field Name	Type	Coverage	Description
Identifiers	url	String	🟢 High (100%)	Original target URL provided in input.
	finalUrl	String	🟢 High (100%)	Resolved canonical URL after HTTP redirects.
	title	String	🟢 High (100%)	Extracted article title or headline.
Authorship	author	String	🟢 High (98%)	Author byline (cleaned of "By", timestamps, or social handles).
	siteName	String	🟢 High (95%)	Brand or publication name hosting the article.
	lang	String	🟢 High (100%)	ISO language code detected from document attributes.
Timestamps	publishedDate	String	🟢 High (90%)	ISO 8601 publication date.
	modifiedDate	String	🟡 Conditional	ISO 8601 last modified date when published.
	extractedAt	String	🟢 High (100%)	Extraction timestamp recorded by the actor.
Content Body	excerpt	String	🟢 High (98%)	Summary snippet or meta description.
	markdown	String	🟢 High (100% opt)	Clean ATX-formatted Markdown body.
	text	String	🟢 High (100% opt)	Sanitized plain-text body without markup.
	html	String	🟢 High (opt-in)	Cleaned article inner HTML without page wrappers.
Media & Tags	mainImage	String	🟢 High (92%)	Primary feature or OpenGraph header image.
	images	Array	🟢 High (100% opt)	List of all resolved image links found in body.
	keywords	String	🟡 Conditional	Publisher-specified content keywords.
Metrics & Errors	wordCount	Integer	🟢 High (100%)	Word count of the extracted clean body.
	readingTimeMinutes	Integer	🟢 High (100%)	Estimated reading duration in minutes (based on 230 wpm).
	error	String	🟡 Conditional	Populated only if the page returned an HTTP error or zero content.

---

🔁 Batch Scale & High-Volume Processing

The extractor is designed for large-scale production workflows:

• Zero Result Caps: Feed hundreds or thousands of links via startUrls with maxItems: 0 for unrestricted batch extraction.
• Deduplication Engine: Automatically removes redundant input URLs, ensuring each unique article is fetched and charged only once.
• Fail-Safe Processing: Transient network or publisher timeouts output an explicit error record rather than crashing the execution batch.
• Resource Efficiency: Consumes under 500 MB RAM even when parsing dense 20,000-word encyclopedic pages.

---

🛠️ Step-by-Step Walkthrough & Integrations

1. Provide URLs: Paste individual URLs into the Article URLs field, upload a .csv or .txt file, or link an external Google Sheet.
2. Choose Output Formats: Select whether you require Markdown, plain text, cleaned HTML, or image lists.
3. Execute Extraction: Click Start. Live logs will track the processing rate and successful extractions.
4. Download or Stream: Download results as JSON, CSV, or Markdown files, or ingest via REST API.

Programmatic Python Integration (with LangChain RAG)

from apify_client import ApifyClient
from langchain_core.documents import Document

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "startUrls": [
        {"url": "https://blog.apify.com/what-is-web-scraping/"},
        {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}
    ],
    "includeMarkdown": True,
    "includeText": False
}

run = client.actor("haketa/article-content-extractor").call(run_input=run_input)
items = client.dataset(run["defaultDatasetId"]).list_items().items

# Convert extracted articles directly into LangChain Documents
documents = [
    Document(
        page_content=item["markdown"],
        metadata={
            "title": item["title"],
            "url": item["url"],
            "author": item.get("author"),
            "publishedDate": item.get("publishedDate")
        }
    )
    for item in items if not item.get("error")
]

print(f"Created {len(documents)} clean LangChain documents ready for vector embedding.")

Programmatic Python Integration (with LlamaIndex)

from apify_client import ApifyClient
from llama_index.core import Document

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("haketa/article-content-extractor").call(run_input={
    "startUrls": [{"url": "https://en.wikipedia.org/wiki/Vector_database"}],
    "includeMarkdown": True
})
items = client.dataset(run["defaultDatasetId"]).list_items().items

llama_docs = [
    Document(
        text=item["markdown"],
        extra_info={
            "title": item["title"],
            "source": item["url"],
            "word_count": item["wordCount"]
        }
    )
    for item in items if not item.get("error")
]
print(f"Prepared {len(llama_docs)} LlamaIndex documents.")

RAG Chunking & Ingestion Architecture

When using the extracted Markdown in vector search pipelines:

1. Markdown Header Splitter: Use header-aware text splitters (MarkdownHeaderTextSplitter) to split on #, ##, and ### headers. This preserves semantic section boundaries.
2. Contextual Metadata: Store the author, publishedDate, siteName, and url within each chunk's metadata dictionary for post-retrieval filtering.
3. Token Efficiency: Clean Markdown consumes approximately 40–70% fewer tokens than raw HTML, reducing embedding generation costs and preventing context-window exhaustion.

Programmatic Node.js Integration

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('haketa/article-content-extractor').call({
    startUrls: [
        { url: 'https://en.wikipedia.org/wiki/Artificial_intelligence' }
    ],
    includeMarkdown: true,
    includeImages: true,
    maxConcurrency: 5
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Extracted article: "${items[0].title}" (${items[0].wordCount} words)`);

Webhook & Automation Workflow

Configure an automated Apify Webhook on ACTOR.RUN.SUCCEEDED to push extracted Markdown documents into vector stores (Pinecone, Qdrant, ChromaDB) or trigger downstream summaries via Zapier or Make.com:

{
  "eventType": "ACTOR.RUN.SUCCEEDED",
  "eventData": {
    "actorId": "haketa/article-content-extractor",
    "actorRunId": "5nmK91XpQ82ZaB0",
    "defaultDatasetId": "wQ90aLkMx47Nb12"
  }
}

---

📊 Supported Export Formats & Integrations

• JSON / JSONL: Optimized for developers, API ingestion, and vector embedding pipelines.
• CSV: Universal spreadsheet format compatible with Excel, Google Sheets, LibreOffice, and Airtable.
• XLSX: Native Microsoft Excel workbook format with column formatting.
• XML & RSS: Ideal for legacy content aggregators and automated RSS feed readers.
• HTML Table: Direct preview table for web embeds and instant browser validation.
• Python & Node.js SDKs: Programmatic fetching using official Apify clients.

---

💡 Best Practices & Pro Tips

• 🎯 Target Article URLs: Point the extractor at specific story or blog post URLs rather than root homepages or category index pages.
• ⚡ Optimize Token Economy: For RAG pipelines, leave includeMarkdown: true and set includeText: false and includeHtml: false to minimize payload sizes.
• 🧹 Filter Non-Articles by Word Count: Post-filter results to drop items where wordCount < 150 to automatically exclude paywalled or landing pages.
• 🔗 Pair with Feed Readers: Pair with an RSS Feed Reader actor to discover new URLs automatically, then feed them into Article Content Extractor for automated content archiving.

---

❓ Frequently Asked Questions

Does this actor crawl an entire domain?
No. This is a dedicated, deterministic content extractor. It processes only the specific URLs you provide, ensuring predictable run times and zero runaway crawl costs.

Can I extract thousands of articles in a single run?
Yes. You can supply thousands of URLs via direct input, text files, or linked Google Sheets. The actor automatically deduplicates URLs and executes with parallel concurrency.

How does the extractor remove advertisements and navigation?
The actor uses advanced structural readability algorithms that analyze text density, link ratios, tag semantics, and DOM layouts to isolate the central narrative text while discarding boilerplate elements.

What happens if a URL is behind a login or hard paywall?
If content is inaccessible, the actor records an informative message in the error field rather than failing the run, allowing your batch processing to complete uninterrupted.

Are relative URLs converted to absolute links?
Yes. All image sources and hyperlinks found within the article body are resolved against the document's canonical base URL.

How does language detection work?
The actor inspects top-level document attributes (<html lang="...">), OpenGraph metadata, and JSON-LD publication tags, returning standardized ISO language codes (e.g., en, fr, de, es).

Can I run this actor continuously on a schedule?
Yes. Configure an Apify schedule (hourly, daily, weekly) or trigger runs programmatically via webhooks from your CMS or ingestion pipeline.

Does the extractor preserve source code snippets?
Yes. Fenced code blocks (```python ... ```) and inline <code> snippets in technical articles are preserved in the Markdown output.

---

📋 Changelog

v1.2 (2026-11)

• 🚀 Unlocked 1,000+ Batch Scale: Added flexible batch handling and maxItems quota governance.
• ⚡ Zero-Input Prefill Fallback: Empty test runs now gracefully fall back to sample articles with zero manual configuration.
• 🧼 Enhanced HTML-to-Markdown Pipeline: Refined Turndown rules to preserve code blocks, blockquotes, and tables accurately.
• 🛡️ Strict Schema Compliance: Validated Input, Output, and Dataset schemas to meet 100% Apify specifications.

v1.1 (2026-08)

• Migrated DOM parsing engine for 3.7x faster throughput and reduced memory consumption.
• Implemented absolute URL resolution for inline body images and relative hyperlinks.

v1.0 (2026-08)

• Initial release featuring clean plain text, structured Markdown, and multi-source metadata extraction.