# Website to Markdown for LLM — Crawl Website to Markdown (`nexgendata/website-content-crawler`) Actor

Website to markdown for LLM use: crawl website to markdown, text or HTML — entire sites, JS-rendered pages included. A website to markdown crawler for RAG pipelines, LLM context and AI training. Firecrawl & Jina alternative. Pay per page.

- **URL**: https://apify.com/nexgendata/website-content-crawler.md
- **Developed by:** [NexGenData](https://apify.com/nexgendata) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Features:** Limited permissions
- **Stats:** 41 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown for LLM — Website Crawler for RAG

Website to Markdown is a NexGenData tool on Apify that returns page content as Markdown, text or HTML from the websites you crawl, including JavaScript-rendered pages.

**Website to markdown for LLM** context: point it at a docs site and get each page as clean markdown, ready to chunk for RAG.

A **website to markdown crawler**: start from a URL and get every page back as clean markdown or text — a **website crawler for RAG** and LLM pipelines.

Full-Site Scraping for AI

> Public web content, for research and RAG.

### 📊 What you get

Clean JSON output (16 fields):

- `url` — Page URL
- `statusCode` — HTTP status of the page
- `title` — Page title
- `description` — Meta description
- `author` — Author, when the page states one
- `publishedDate` — Published date, when the page states one
- `modifiedDate` — Last-modified date, when the page states one
- `siteName` — Site name
- `language` — Page language
- `content` — Main content as plain text
- `html` — Cleaned HTML of the main content
- `markdown` — Main content as Markdown (for LLM/RAG use)
- `links` — Links found on the page (URL and link text)
- `images` — Images in the content (URL and alt text)
- `wordCount` — Words in the main content
- `crawledAt` — When the page was read (UTC)

**Pricing:** $0.020 per page (Pay-Per-Event) — about **50 pages per $1**.

### ⚙️ Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `start_urls` | \['array', 'string'] | `["https://example.com/"]` | Start URLs for the website to markdown crawler. URLs to start crawling from. Accepts a list of URLs OR a single URL string OR a comma/newline-separated list. Also accepts these alternative field names: urls, startUrls, c |
| `max_pages` | integer | `10` | Maximum number of pages to crawl per start URL |
| `max_depth` | integer | `2` | Maximum crawl depth (0 = start URL only, 1 = start URL + linked pages, etc.) |
| `same_domain` | boolean | `true` | Only crawl pages on the same domain as the start URL |
| `exclude_patterns` | array | `["/admin", "/login", "/wp-login"]` | URL patterns to exclude from crawling (e.g., /admin, /login) |
| `output_format` | string | `"json"` | Format for extracted content |
| `urls` | \['array', 'string'] | `""` | Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated. |
| `startUrls` | \['array', 'string'] | `""` | Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated. |
| `crawlUrls` | \['array', 'string'] | `""` | Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated. |
| `pages` | \['array', 'string'] | `""` | Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated. |
| `url` | string | `""` | Alternative — a single URL when you only need one. Comma-separated values are auto-split. |

### ▶️ Example input

```json
{
  "start_urls": [
    "https://example.com/"
  ],
  "exclude_patterns": [
    "/admin",
    "/login",
    "/wp-login"
  ]
}
```

### 🧾 Example output (one real row)

```json
{
  "url": "https://example.com/",
  "statusCode": 200,
  "title": "Example Domain",
  "description": null,
  "author": null,
  "publishedDate": null,
  "modifiedDate": null,
  "siteName": null,
  "language": "en",
  "content": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
  "html": "<body><p>This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.</p></body>",
  "markdown": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
  "links": [],
  "images": [],
  "wordCount": 25,
  "crawledAt": "2026-11-03T20:00:30.692216Z"
}
```

### 🤖 Use with AI agents

Point Claude, the OpenAI Agents SDK, an n8n flow or any MCP-aware client at it.

**Sample agent prompt:**

```
Run Website Crawler on my input and return the structured results.
```

**Agentic payments (x402):** Supports agentic payment via x402 — call this actor with USDC, no API key required.

### 🔗 Related tools — Web crawling for AI/RAG

[RAG Web Browser](https://apify.com/nexgendata/rag-web-browser)

***

*A NexGenData utility actor.*

# Actor input Schema

## `start_urls` (type: `array,string`):

Start URLs for the website to markdown crawler. URLs to start crawling from. Accepts a list of URLs OR a single URL string OR a comma/newline-separated list. Also accepts these alternative field names: urls, startUrls, crawlUrls, pages, url.

## `max_pages` (type: `integer`):

Maximum number of pages to crawl per start URL

## `max_depth` (type: `integer`):

Maximum crawl depth (0 = start URL only, 1 = start URL + linked pages, etc.)

## `same_domain` (type: `boolean`):

Only crawl pages on the same domain as the start URL

## `exclude_patterns` (type: `array`):

URL patterns to exclude from crawling (e.g., /admin, /login)

## `output_format` (type: `string`):

Format for extracted content

## `urls` (type: `array,string`):

Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated.

## `startUrls` (type: `array,string`):

Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated.

## `crawlUrls` (type: `array,string`):

Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated.

## `pages` (type: `array,string`):

Alternative name for `start_urls`. Accepts a list of URLs OR a single URL string OR comma/newline-separated.

## `url` (type: `string`):

Alternative — a single URL when you only need one. Comma-separated values are auto-split.

## Actor input object example

```json
{
  "start_urls": [
    "https://example.com/"
  ],
  "max_pages": 10,
  "max_depth": 2,
  "same_domain": true,
  "exclude_patterns": [
    "/admin",
    "/login",
    "/wp-login"
  ],
  "output_format": "json"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": [
        "https://example.com/"
    ],
    "exclude_patterns": [
        "/admin",
        "/login",
        "/wp-login"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("nexgendata/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "start_urls": ["https://example.com/"],
    "exclude_patterns": [
        "/admin",
        "/login",
        "/wp-login",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("nexgendata/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": [
    "https://example.com/"
  ],
  "exclude_patterns": [
    "/admin",
    "/login",
    "/wp-login"
  ]
}' |
apify call nexgendata/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nexgendata/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mz8p4p7FT3emhdAhe/builds/ePX8TbmXMI3UEenl4/openapi.json