# Website Content Crawler - Markdown, 45 Fields (`parseforge/website-content-crawler`) Actor

Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.

- **URL**: https://apify.com/parseforge/website-content-crawler.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 191 total users, 10 monthly users, 90.7% runs succeeded, 2 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner.jpg)](https://apify.com/parseforge?fpr=vmoqkp)

### Website Content Crawler

**Crawl any website and extract clean text content for your AI models, LLM pipelines, and vector databases.** Every page returns its full Markdown body, a plain-text version, and 45 metadata fields. No sitemap needed. Export to CSV, JSON, Excel, or XML.

Building a RAG pipeline or fine-tuning an LLM requires clean, structured text from websites, but writing a custom crawler for each site is slow and brittle. The Website Content Crawler reads public pages from any starting URL, follows links up to a set depth, and returns the full text content of each page in a consistent format. It handles HTML cleaning, Markdown conversion, and domain filtering so your data is ready for embeddings or training.

| Who uses it | What they scrape Website Content Crawler for |
|---|---|
| AI engineers | Building a knowledge base for a RAG application or fine-tuning dataset. |
| Data scientists | Collecting a corpus of text from a specific domain for NLP analysis. |
| SEO specialists | Extracting all body text from a competitor's site to audit content strategy. |
| Developers | Feeding a vector database with Markdown-formatted pages from documentation sites. |

### What it does

This Actor crawls websites from a list of starting URLs and returns each page's full text content, a Markdown body, and 45 metadata fields as a flat row.

- 📝 **Markdown output:** every page is converted to clean Markdown, ready for LLM context windows.
- 🧹 **HTML cleaning:** navigation, scripts, and boilerplate are stripped, leaving only the main content.
- 🔗 **Link crawling:** follows internal links up to a configurable depth to discover all pages on a site.
- 🌐 **Domain control:** restrict the crawl to the same domain, or optionally include subdomains.
- 🎯 **URL filtering:** include or exclude pages with regex patterns to target only the content you need.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with Website Content Crawler data

**🤖 Build a RAG knowledge base.**

An AI engineer crawls a product documentation site to extract every page as Markdown, then loads the dataset into a vector store for a support chatbot.

**📊 Audit a competitor's content.**

An SEO specialist crawls a competitor's blog, filtering URLs to only article pages, and extracts the full body text to analyze topic coverage and keyword density.

**🧠 Create a fine-tuning dataset.**

A data scientist crawls a public forum, limits the depth to thread pages, and collects a corpus of question-answer pairs for instruction-tuning a model.

**📚 Archive documentation for offline use.**

A developer crawls an open-source library's docs site, stays on the same domain, and exports the full text to a single JSON file for local search.

### Why choose this scraper

|  | What you get |
|---|---|
| **Clean text extraction** | Full page body in Markdown and cleaned HTML, stripped of navigation and scripts. |
| **Configurable crawl scope** | Set max depth, page limit, and domain restrictions to control the crawl size. |
| **Regex URL filtering** | Include or exclude specific URL patterns to target only relevant sections of a site. |
| **AI-ready output** | Output format is designed for direct ingestion into LLMs, vector databases, and LangChain. |

### How it compares

This Actor focuses on extracting clean, AI-ready text from any website, while the other Website Content Crawler listing offers similar Markdown extraction and LangChain integration.

| Feature | ParseForge | Website Content Crawler |
|---|---|---|
| Full page text in Markdown | Yes | Yes |
| Cleaned HTML output | Yes | Yes |
| Regex URL filtering | Yes | Not listed |
| Subdomain inclusion control | Yes | Not listed |
| LangChain integration | Not listed | Yes |

### Configure the run

Drive the Actor from one or more starting URLs, set a crawl depth and page limit, and apply domain and regex filters so only matching pages reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true
}
```

A larger pull:

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 200,
  "sameDomain": true,
  "includeSubdomains": true
}
```

### Pricing

Pay-per-result: **$0.025 per result** collected. You pay only for the results written to your dataset.

| Results collected | Approximate cost |
|---|---|
| 100 results | $2.50 |
| 1,000 results | $25.00 |
| 10,000 results | $250.00 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [Website Content Crawler](https://apify.com/parseforge/website-content-crawler?fpr=vmoqkp).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to Website Content Crawler through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/website-content-crawler"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why am I getting no results?**

Check your Include URL Patterns regex. If it is too restrictive, no discovered URLs will match. Try removing the pattern or testing it against a known page URL. Also verify the starting URL is accessible and returns a 200 status.

**The crawl is taking too long.**

Reduce the maxDepth or maxItems to limit the crawl scope. You can also add Exclude URL Patterns to skip large sections like archives or tag pages that generate many low-value URLs.

**The extracted text contains navigation menus and footer links.**

The Actor cleans common HTML boilerplate, but some sites use unusual markup. The cleaned HTML field removes scripts and styles, but complex layouts may still include some chrome. The Markdown field focuses on the main content area.

**I am getting pages from other domains even with 'Stay on Same Domain' enabled.**

Check if you have 'Include Subdomains' enabled. A subdomain like blog.example.com is treated as a different domain from www.example.com. Disable subdomain inclusion to restrict to the exact starting domain.

**The Markdown output is missing formatting like headings or lists.**

The Markdown conversion preserves semantic HTML elements like h1-h6, ul, ol, and p tags. If a site uses divs with CSS classes instead of semantic tags, the structure may be lost. The cleaned HTML field will still contain the original structure.

### FAQ

| Question | Answer |
|---|---|
| What format is the extracted text in? | Each page returns a Markdown version of the main content and a cleaned HTML version. The Markdown is ready for direct use with LLMs and vector databases. |
| Can I limit the crawl to only one section of a website? | Yes. Use the Include URL Patterns field with a regex like `/docs/.*` to crawl only pages under the /docs/ path, or use Exclude URL Patterns to skip sections like /blog/. |
| Does it handle JavaScript-rendered content? | This Actor fetches the initial HTML of each page. For sites that load content dynamically with JavaScript, you may need a headless browser crawler instead. |
| How many pages can I crawl in one run? | You can set the maximum pages up to 1,000,000 per run. The actual number depends on your max depth, domain restrictions, and URL filters. |
| Will it follow links to external websites? | Not by default. The 'Stay on Same Domain' option is on by default, which restricts the crawl to the starting domain. You can disable it, but the Actor will still only follow links it discovers from the pages it crawls. |
| Can I use this to scrape a single page? | Yes. Set maxDepth to 0 and maxItems to 1, and it will extract only the content from the starting URL without following any links. |
| What is the output schema? | Each row in the dataset includes the page URL, the full Markdown body, the cleaned HTML, the page title, and metadata like the crawl depth and load timestamp. |
| Does it download images or PDFs? | No, this Actor extracts text content only. It does not download binary files, images, or PDFs. |
| Can I crawl multiple websites in one run? | Yes. Add multiple starting URLs from different domains. If 'Stay on Same Domain' is on, each starting URL will spawn a separate crawl restricted to its own domain. |
| Is this suitable for a production RAG pipeline? | Yes. The Markdown output is designed for direct ingestion into vector databases like Pinecone or Weaviate, and the Actor integrates with LangChain for loading documents. |

### Related actors

Browse the full [ParseForge collection](https://apify.com/parseforge?fpr=vmoqkp) for more scrapers.

⭐ **Found this Actor useful?** [Leave a review](https://console.apify.com/actors/XvOSmmcClKGzwlQOJ/reviews). It takes a minute and helps us keep it maintained.

🆘 **Need help?** Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Apify. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `startUrls` (type: `array`):

Starting URLs to crawl.

## `maxDepth` (type: `integer`):

How many link hops from start URLs.

## `maxItems` (type: `integer`):

How many pages to collect per run.

## `sameDomain` (type: `boolean`):

Restrict crawl to the starting domain

## `includeSubdomains` (type: `boolean`):

Allow subdomains when crawling

## `includeUrlPatterns` (type: `array`):

Regex patterns to include URLs

## `excludeUrlPatterns` (type: `array`):

Regex patterns to exclude URLs

## `renderingType` (type: `string`):

How pages are fetched. HTTP is the fastest and cheapest and works for most sites. Auto starts with HTTP and only renders a page in a browser when HTTP comes back empty or blocked. Browser renders every page, which is much slower and more expensive.

## `useSitemap` (type: `boolean`):

Look for the site's sitemap.xml and use it to find pages that are not linked from the start URL.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site's robots.txt disallows for crawlers. Turn this off only for a site you are authorised to crawl.

## `proxyConfiguration` (type: `object`):

Proxy used for every request. Many sites block datacenter IPs outright, so leaving this on is what keeps a crawl from coming back empty. Each retry goes out on a fresh session, so a page blocked on one IP is retried on another. Residential costs more and is only needed for the hardest sites.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true,
  "renderingType": "http",
  "useSitemap": true,
  "respectRobotsTxt": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Complete dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com/docs"
        }
    ],
    "maxDepth": 2,
    "maxItems": 10,
    "sameDomain": true,
    "includeSubdomains": true,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com/docs" }],
    "maxDepth": 2,
    "maxItems": 10,
    "sameDomain": True,
    "includeSubdomains": True,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call parseforge/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XvOSmmcClKGzwlQOJ/builds/acEuzb5n34P2qPZBC/openapi.json