# Website to Markdown – Clean LLM & RAG Content Extractor (`dataquarry/website-to-markdown`) Actor

Convert any public web page to clean, LLM-ready Markdown with metadata — by URL, a list of URLs, or a whole-site crawl. Strips nav/ads/boilerplate, keeps headings/lists/tables/code. Respects robots.txt. No API key.

- **URL**: https://apify.com/dataquarry/website-to-markdown.md
- **Developed by:** [Daniel Brenner](https://apify.com/dataquarry) (community)
- **Categories:** Developer tools, AI, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown — Clean LLM & RAG Content Extractor

**Convert any public web page to clean, LLM-ready Markdown — by URL, a list of URLs, or a whole-site crawl.** No API key, no browser to manage. The tool fetches the page, strips navigation, ads and boilerplate, and returns tidy Markdown plus structured metadata — ideal for feeding LLMs, building RAG pipelines, archiving articles, or training datasets.

It's the no-fuss answer to "**how do I turn a website into Markdown for ChatGPT / a vector database / an AI agent?**" — give it URLs, get Markdown back as JSON, CSV or Excel.

### What it does

- **HTML → clean Markdown**, with headings, lists, tables, code blocks and links preserved (GitHub-flavored).
- **Main-content extraction** — removes menus, headers, footers, sidebars, cookie banners and ads so the Markdown is just the article, not the chrome. (Or keep the full page if you prefer.)
- **Structured metadata** per page — title, description, author, published date, language, site name, canonical URL.
- **RAG-ready extras** — optional token-count estimate and overlapping chunks for retrieval-augmented generation.
- **Three modes** — a single page, a list of pages, or crawl a site (same-domain link following, depth- and page-limited).
- **Respects `robots.txt`** and sends a descriptive User-Agent. Public pages only — no logins, no paywalls.

### What you get (per page)

| field | description |
|-------|-------------|
| `url` / `finalUrl` | requested URL, and the final URL after redirects (if different) |
| `title`, `description`, `author`, `publishedDate`, `lang`, `siteName` | page metadata (honest-null when not present) |
| `canonicalUrl` | the page's canonical link, if declared |
| `markdown` | the clean, LLM-ready Markdown |
| `wordCount`, `tokenEstimate` | length, plus an **approximate** LLM token count (a heuristic estimate, not exact) |
| `chunks` | optional RAG chunks (`{index, text}`) when "Chunk for RAG" is on |
| `links`, `images` | optional lists of absolute links / image URLs on the page |
| `httpStatus`, `fetchedAt`, `robotsAllowed` | response status, fetch time, robots check result |

Anything a page doesn't expose comes back as `null` — never guessed.

### How to use it

**Single page or a list:**

```json
{ "startUrls": ["https://example.com/blog/post"], "mode": "single" }
```

```json
{ "startUrls": ["https://a.com/p1", "https://a.com/p2"], "mode": "list" }
```

**Crawl a site for a RAG dataset:**

```json
{
  "startUrls": ["https://docs.example.com/"],
  "mode": "crawl",
  "maxCrawlDepth": 2,
  "maxPages": 100,
  "sameDomainOnly": true,
  "chunkForRag": true,
  "chunkSize": 2000,
  "chunkOverlap": 200
}
```

Set `contentMode` to `main` (default — just the article) or `full` (the whole page). Toggle `includeLinks` / `includeImages` to also collect a page's links and images.

### Why this tool

- **No API key, no headless browser to babysit** — give it URLs, get Markdown.
- **Clean output** — boilerplate-stripped main content, not a raw HTML dump.
- **LLM-first** — Markdown + metadata + optional chunking and token estimate, the exact shape RAG and fine-tuning pipelines want.
- **Polite & transparent** — respects `robots.txt`, identifies itself, fetches only public pages one at a time.

Pricing is **pay-per-result: $2 per 1,000 pages** — you only pay for pages successfully converted. Export as JSON, CSV or Excel.

### Responsible use

This tool reformats the public pages you point it at — the same content a browser would load. You're responsible for how you use content from sites you don't own (respect each site's terms and applicable copyright). The tool honors `robots.txt` and never bypasses logins or paywalls.

### FAQ

**Do I need an API key?**

No. Give it one or more URLs and run it — no key, no quota.

**Can it convert a whole website, not just one page?**

Yes — use `crawl` mode with a start URL. It follows same-domain links up to the depth and page limits you set, converting each page to Markdown.

**Is the Markdown ready for an LLM / RAG?**

Yes — that's the point. You get clean Markdown plus metadata, an approximate token count, and optional overlapping chunks for retrieval-augmented generation.

**How much does it cost?**

Pay-per-result: $2 per 1,000 pages — you only pay for the pages you actually get.

***

⭐ **Found this useful?** A quick rating on this actor's Store page helps others discover it — and if something is off or you wish it did more, open an issue on the actor. I read every one.

# Actor input Schema

## `startUrls` (type: `array`):

Web page URLs to convert to Markdown. In crawl mode these are the starting pages.

## `mode` (type: `string`):

single = first URL only; list = every URL given; crawl = follow same-domain links from each start URL.

## `contentMode` (type: `string`):

main = extract the primary article/content and strip nav, ads and boilerplate; full = convert the whole page.

## `maxResults` (type: `integer`):

Maximum number of pages to return.

## `maxCrawlDepth` (type: `integer`):

Crawl mode only: how many link-hops from each start URL to follow.

## `maxPages` (type: `integer`):

Crawl mode only: hard cap on pages visited.

## `sameDomainOnly` (type: `boolean`):

Crawl mode only: only follow links on the same domain as the start URL.

## `includeLinks` (type: `boolean`):

Also output the list of links found on each page.

## `includeImages` (type: `boolean`):

Also output the list of image URLs found on each page.

## `computeTokenEstimate` (type: `boolean`):

Add an approximate LLM token count for each page's Markdown (a heuristic estimate, not exact).

## `chunkForRag` (type: `boolean`):

Also split each page's Markdown into overlapping chunks for retrieval-augmented generation.

## `chunkSize` (type: `integer`):

Target chunk size in characters when chunking for RAG.

## `chunkOverlap` (type: `integer`):

Overlap between consecutive chunks when chunking for RAG.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by the site's robots.txt for our user-agent.

## `requestDelayMs` (type: `integer`):

Optional politeness delay between pages, in milliseconds (pages are fetched one at a time).

## Actor input object example

```json
{
  "startUrls": [
    "https://en.wikipedia.org/wiki/OpenStreetMap"
  ],
  "mode": "list",
  "contentMode": "main",
  "maxResults": 50,
  "maxCrawlDepth": 1,
  "maxPages": 50,
  "sameDomainOnly": true,
  "includeLinks": false,
  "includeImages": false,
  "computeTokenEstimate": true,
  "chunkForRag": false,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "respectRobotsTxt": true,
  "requestDelayMs": 0
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://en.wikipedia.org/wiki/OpenStreetMap"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("dataquarry/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://en.wikipedia.org/wiki/OpenStreetMap"] }

# Run the Actor and wait for it to finish
run = client.actor("dataquarry/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://en.wikipedia.org/wiki/OpenStreetMap"
  ]
}' |
apify call dataquarry/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dataquarry/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Zpoaapkb6FbIaeBiJ/builds/8ioY3ribmeJmRR22B/openapi.json