# Website to Markdown Crawler - Content Crawler for LLM & RAG (`tidytools/website-markdown-crawler`) Actor Website to Markdown crawler for RAG and LLMs: crawl a website or docs site into clean Markdown (URL to Markdown, HTML to Markdown) with RAG chunks. Sitemaps, JavaScript pages, linked PDFs and Word. $1/1k pages. - **URL**: https://apify.com/tidytools/website-markdown-crawler.md - **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community) - **Categories:** AI, Developer tools, Automation - **Features:** Limited permissions - **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.80 / 1,000 page (fast mode)s This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Website to Markdown Crawler do? **Website to Markdown Crawler converts a website to Markdown for RAG: give it a start URL (a docs site, help center or blog) and get clean, LLM-ready Markdown for every page, RAG chunks with heading paths and one combined `OUTPUT.md`, for $1 per 1,000 pages ($2.50 when a real browser is needed), compute included. Failed, blocked and duplicate pages are free.** #### How to convert a website to Markdown for RAG 1. Enter the site in **Start URLs**, one per line (for example `https://docs.example.com/` or just `docs.example.com`). A new Console input is prefilled with **Max pages: 10** for a small first run; you can raise **Max pages** yourself. If an API input omits `maxPages`, the default is still 50. 2. Optional: set **RAG chunk size (characters)**, for example 2000. 3. Click **Start**. Each page becomes one row with `markdown` and `chunks`; `OUTPUT.md` holds the whole site in one file. A page that cannot be read gets a free row with an `error` and a `hint` saying what to try (see [FAQ](#faq)). 4. Load the chunks into your vector store (LangChain and LlamaIndex examples below) or fetch them via API. A 1,000-page docs site costs $1 in fast mode. #### Why this one - **A price you know before the run.** Apify's free Website Content Crawler (WCC) bills compute time plus Apify Proxy instead: its README estimates about $0.20 per 1,000 pages over plain HTTP and $0.50–$5 with a browser (checked September 2026), and its default settings cost $1.55–$3.65 per 1,000 pages in our benchmark. A tuned WCC run can still cost less on large plain-HTTP sites ([comparison](#compared-with-apifys-website-content-crawler)). - **Keeps the body text.** In our 5-site test (25 pages each, default settings) this Actor kept at least 97% of the sampled body paragraphs on every site and took 3–34 seconds per site; WCC took 56–200 seconds and kept 0% on a Next.js docs site and 34% on an Intercom help center ([benchmark](https://tools.yukai.uk/benchmarks/markdown-vs-wcc)). - **Linked PDF, Word, Excel and CSV files are converted** to Markdown (WCC can save the files, but does not convert them). - **RAG chunks with heading paths** and one combined `OUTPUT.md` file (llms-full.txt style) in the same run, at no extra charge. - **No proxy or API key to set up**: pages are fetched from our servers first, and a real browser is used only when a page needs it. #### Input (copy and paste) ```json { "urls": ["https://docs.apify.com/academy"], "maxPages": 10, "chunkSize": 2000 } ``` #### Output (real result, shortened) One page from a test run on `docs.apify.com/academy`, read in fast mode and charged $0.001. For the 10-page limit in the first-run input above, 10 successfully converted pages cost at most $0.01 if all use fast mode, or at most $0.025 if all use browser mode. Actual cost depends on the number of successful pages and each page's mode; pages that fail to convert are free. The crawl may find fewer than 10 pages: ```json { "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description", "startUrl": "https://docs.apify.com/academy", "title": "Actor description & SEO description | Academy | Apify Documentation", "mode": "fast", "success": true, "charged": true, "wordCount": 1205, "markdown": "# Actor description & SEO description\n\nCopy for LLM\n\nLearn about Actor description and meta description. …\n\n## What is an Actor description?\n\n…", "contentDigestKind": "markdown:v1", "contentDigest": "sha256:…", "chunks": [ { "index": 1, "text": "… ### Is there any benefit in the description and meta description being different?\n\n…", "headingPath": ["Actor description & SEO description", "Regular description vs. SEO description", "Is there any benefit in the description and meta description being different?"], "charCount": 1850 } ], "crawledAt": "2026-09-29T03:56:19.278Z" } ``` All fields, the failed-page row and the `SUMMARY` record: [Output example](#output-example). ### How much does it cost? Pay per event, only for pages that were converted successfully: | Event | Price | |---|---| | Page (fast mode, plain HTTP) | $1.00 / 1,000 pages | | Page (browser mode, JavaScript rendering) | $2.50 / 1,000 pages | **Example:** a 2,000-page documentation site costs **$2.00** when every page is read in fast mode, and at most **$5.00** if every page needed the browser. **No start fee.** Compute time is included: the table is the whole price. In **Auto** mode (default) most pages use the fast mode. Duplicates, blocked pages (403, bot checks), pages disallowed by robots.txt, pages with no text even after the browser fallback, and errors are not charged. Your maximum charge limit is always respected. On Apify's Scale plan prices are 10% lower, on Business and higher 20% lower ($0.80 / 1,000 fast pages). ### Features - 📄 **Documents too**: linked **PDF, Word (DOCX), Excel (XLSX) and CSV** files are converted to Markdown, not just downloaded - ✂️ **RAG chunks built in**: optional `chunks` array split at paragraph boundaries with overlap, each with its `headingPath` (h1 > h2 > h3) and `charCount`, ready for embeddings - 📚 **One-file export**: every converted page also goes into one `OUTPUT.md` file in crawl order (llms-full.txt style), each page under a `# Title` header with a `Source:` line, ready to paste into an LLM. Its link is `combinedFileUrl` in the `SUMMARY` record. Free, on by default - 💵 **Fixed price per page**: **$1 per 1,000 pages** over plain HTTP, $2.50 when a real browser is needed. Compute is included, and failed, blocked and duplicate pages are **free** - 🧹 **Clean Markdown by default**: menus, footers, sidebars, related-post and newsletter blocks are removed; code fences, tables and absolute links are kept. Add your own CSS selectors to remove more, or keep only one part of the page (details under **What gets removed** in Tips) - 🌐 **Many sites in one run**: paste start URLs one per line (bare domains are fine). Every page has `startUrl`, so you can group results by site, **Max pages per start URL** gives each site its share, and the `SUMMARY` record has one entry per site - 🕷️ **Crawls for you**: start from one URL and follow links within the same folder, the whole site, or only the pages you list; the site's `sitemap.xml` is read too, so pages behind JavaScript menus are not missed. Links inside page content are crawled first; other languages, old versions and author/tag listing pages come last, so a small **Max pages** goes to real content - 🤖 **Respects robots.txt** by default (can be turned off for your own sites) - 🏷️ **Page metadata**: `description`, `language`, `canonicalUrl` and `wordCount` for every page - 🔤 **Markdown or plain text**: choose one or both - ⚡ **Real browser only when needed**: JavaScript-only pages and sites that block simple requests are rendered in headless Chrome automatically ### Who is it for? - Building a **RAG chatbot** or **AI assistant** over your docs, help center or website - Feeding **LLM agents** with up-to-date documentation - Creating a **knowledge base export** or content audit - **Migrating** a website's content to another CMS ### Price comparison with other crawlers #### Compared with Apify's Website Content Crawler Apify's free **Website Content Crawler** (WCC) is the most used crawler on the Store and has more options than this Actor (several HTML-to-Markdown converters, logins, expanding hidden content, HTML output). You pay for its compute time instead of per page. Its own README estimates about **$0.20 per 1,000 pages** with the plain HTTP crawler and **$0.50–$5 per 1,000 pages** with a browser, depending on the site, plus Apify Proxy (checked September 2026). **Our benchmark** (5 sites, 25 pages each, both tools on the settings a new user gets, September 2026): this Actor took 3–34 seconds per site and WCC 56–200 seconds; WCC's default run cost about $1.55–$3.65 per 1,000 pages in platform usage on our account. WCC's defaults lost most of the body text on a Next.js docs site (0 of 82 sampled paragraphs) and an Intercom help center (34%); this Actor kept at least 97% on all five sites. WCC was still cleaner on blog pages (fewer leftover footer and related-post lines). Method and per-run numbers: [benchmark page](https://tools.yukai.uk/benchmarks/markdown-vs-wcc). So for large sites that work over plain HTTP, a tuned WCC run can cost less than this Actor. Choose this Actor when you want: - a **price you know before the run**: $1 or $2.50 per 1,000 pages, and nothing for failed, blocked or duplicate pages; - **linked PDF, Word and Excel files converted** to Markdown (WCC can save the files, but does not convert them); - **RAG chunks with heading paths** in the same run. Already crawled a site with WCC? [Website to Vector Embeddings](https://apify.com/tidytools/website-to-embeddings) can embed its dataset directly. Other paid crawlers on the Apify Store, per 1,000 pages (checked September 2026): worshipful_knife Website Content Crawler $1.00 (Business plan $0.70) plus platform usage and a start fee; joyouscam35875 Website Content Crawler free today, $2.00 from October 5, 2026; datascoutapi Website Content Crawler Pro $2.99; easyapi Website Content to Markdown $2.99 plus a $0.09 start fee and platform usage; 6sigmag Fast Website Content Crawler $3.00 plus a start fee; parseforge Website Content Crawler $20–$25. | | This Actor | |---|---| | Fixed price per page, compute included | ✅ | | PDF, Word and Excel links converted | ✅ | | RAG chunks with heading paths | ✅ | | Navigation and footer removal | ✅ | | JavaScript pages (real browser when needed) | ✅ | | Sitemap seeding and robots.txt | ✅ | | Duplicates and failed pages free | ✅ | ### Control your cost - **What is charged:** each page converted successfully, at the fast or the browser price. - **What is free:** failed and blocked pages, pages with no text (`errorType: "no_text"`), duplicate pages, pages disallowed by robots.txt and invalid input lines. Every row has `charged: true` or `false`. - **Max pages** caps the whole run; with several start URLs, **Max pages per start URL** stops one big site from using it all. - At the start, the run shows an estimate in the log and the run's status message, e.g. `Estimate: up to 50 page(s) x $0.001 (fast) to $0.0025 (browser) per page = at most $0.125 (about $0.05 if pages read in fast mode). You pay only for pages actually returned; failed, empty and duplicate pages are free.` - It also logs its worst case, e.g. `Plan: 50 pages × $0.0025 = at most $0.125` (every page at the browser price), and warns when that is more than your **maximum charge per run**. - The log also shows the expected cost, and the `SUMMARY` record has both: `maxCostUsd` (the most this run can cost: every planned page at the browser price, or the fast price in Fast mode, capped at your maximum charge per run) and `expectedCostUsd` (every planned page in fast mode, the usual case). - When the maximum charge per run is reached, the run stops and keeps all pages converted so far. The status message says so, and the `SUMMARY` record has `status: "LIMIT_REACHED"` and `notProcessed` (how many queued pages were not crawled, and up to 100 of them). ### From a test run to the whole site Start with a small run (10-20 pages). If it converts at least one page and stops only because it reached **Max pages** or **Max pages per start URL** while more pages were found, the `SUMMARY` record in the run's key-value store has a `nextStep` object. For a 20-page run with a $5 maximum charge per run in Auto mode it looks like this: ```json { "nextStep": { "message": "Small crawl succeeded (20 pages converted). To scale up, set Crawl scope to \"Whole website\", raise Max pages to 2000; estimated cost at most $5 for up to 2000 pages.", "suggestedInput": { "crawlScope": "domain", "maxPages": 2000 }, "estimatedCostUsd": 5 } } ``` - `suggestedInput` lists only the fields to change: `crawlScope` set to `domain` (Whole website), `maxPagesPerStartUrl` set back to `0` (no per-start-URL limit) if you had set one, and `maxPages` set to the number of pages your budget covers. Copy these values into a new run. - `estimatedCostUsd` is the most the new run can cost: Auto and Browser mode count every page at the browser price ($2.50 per 1,000), Fast mode at the fast price ($1 per 1,000). If you set a maximum charge per run, `maxPages` is kept within it; without one, `maxPages` is 10,000 (the highest allowed), so the estimate is $25 in Auto mode. You pay only for pages actually converted; failed, blocked and duplicate pages are free. - There is no `nextStep` when nothing was left to crawl, when no page was converted, when the run stopped because of the memory or maximum charge limit, or when the suggested values would not change your input. ### How to use it 1. Enter one or more **Start URLs**, one per line, e.g. `https://docs.example.com/` or just `docs.example.com`. 2. Choose the **Crawl scope** (same folder is best for documentation). 3. Set **Max pages**. 4. Optional: keep **Remove navigation, headers and footers** on (the default), set a **Content selector** such as `main` or `article`, URL include/exclude patterns, **Output formats**, and a **RAG chunk size** (e.g. 2000). 5. Optional: **Use the site's sitemap** is on (*Seed*) by default; choose *Only* to convert just the sitemap's pages, or *Off* to follow links only. 6. Click **Start** and download the results as JSON, CSV or Excel, or fetch them via API. #### Input example ```json { "urls": ["https://docs.apify.com/academy"], "crawlScope": "path", "maxPages": 200, "maxPagesPerStartUrl": 0, "mode": "auto", "removeBoilerplate": true, "removeSelectors": ".related-posts, #comments", "excludeUrlPatterns": ["**/changelog/**"], "useSitemap": "seed", "respectRobotsTxt": true, "outputFormats": ["markdown", "text"], "chunkSize": 2000, "chunkOverlap": 200 } ``` #### Output example Real output from a test run on `https://docs.apify.com/academy` with **Remove navigation, headers and footers** on, plain text and 2,000-character chunks (long text shortened with …): ```json { "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description", "finalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description", "startUrl": "https://docs.apify.com/academy", "title": "Actor description & SEO description | Academy | Apify Documentation", "depth": 0, "httpStatus": 200, "contentType": "text/html; charset=utf-8", "mode": "fast", "via": "backend", "success": true, "charged": true, "description": "Learn about Actor description and meta description. Where to set them and best practices for both content and length.", "language": "en", "canonicalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description", "wordCount": 1205, "foundIn": "sitemap", "markdown": "# Actor description & SEO description\n\nCopy for LLM\n\nLearn about Actor description and meta description. …\n\n## What is an Actor description?\n\n…", "text": "Actor description & SEO description\n\nCopy for LLM\n\nLearn about Actor description and meta description. …", "contentDigestKind": "markdown:v1", "contentDigest": "sha256:…", "chunks": [ { "index": 1, "text": "… ### Is there any benefit in the description and meta description being different?\n\n…", "headingPath": ["Actor description & SEO description", "Regular description vs. SEO description", "Is there any benefit in the description and meta description being different?"], "charCount": 1850 } ], "crawledAt": "2026-09-29T03:56:19.278Z" } ``` Without the clean-up option, pages on this site start with `[Skip to main content](#…)` and `On this page`, and end with the page's table-of-contents links. On `docs.apify.com/academy/web-scraping-for-beginners` the option cut the Markdown from 7,612 to 7,108 characters by removing exactly those parts. - `via` says where the page was fetched from: `backend` (our servers), `direct` (Apify's network or your proxy) or `browser`. - `foundIn` is `start`, `sitemap` or `link`. - `startUrl` is the start URL the page was reached from. `url` is the address requested, kept as you entered it for start pages (a trailing `/` is not removed); start pages also have `inputUrl`, the line exactly as you typed it. - `description`, `language` and `canonicalUrl` are `null` when the page does not declare them (and may be `null` for pages rendered in the browser). - `text` appears only when **Output formats** includes *Plain text*; `markdown` is `null` if you leave Markdown out. - `contentDigestKind` and `contentDigest` are a SHA-256 digest of this row's own output text, so you can tell whether two rows carry the same content, or diff two crawls of the same site, without re-reading the pages. `contentDigest` is `sha256:` plus the hex digest of the UTF-8 bytes of `\0`, with no other normalisation: the same text always gives the same digest, and one changed character gives a different one. `contentDigestKind` is `markdown:v1` (the digest is of the `markdown` field) when Markdown output is on, and `text:v1` (the `text` field) when you leave Markdown out; when the row has both, the digest is of `markdown`. - The digest is of this row's own output text only — not of `OUTPUT.md`, the page's original HTML, or the fetch time — so it changes only when the row's content changes. Failed pages and pages with no text have no digest fields (see the failed-page row below). - Tip: turn on **Remove navigation, headers and footers**, or set **Content selector** to `main` or `article`, to drop "Skip to main content" and menu text. A failed page is not charged and looks like this: ```json { "url": "https://example.com/missing", "startUrl": "https://example.com/", "depth": 1, "success": false, "error": "target returned HTTP 404 (not charged)", "errorType": "not_found", "hint": "The page does not exist (HTTP 404/410). Check the start URL for typos, …", "routesTried": ["backend"], "charged": false } ``` The run's key-value store has a `SUMMARY` record with `status` (`SUCCESS`, `PARTIAL_RESULTS`, `FAILED`, `NO_RESULTS`, `LIMIT_REACHED` or `INVALID_INPUT`), page counts, `disallowedByRobotsTxt`, `sitemapUrlsQueued`, `emptyPagesNotCharged`, `sites` (one entry per start URL: `pages`, `failed`, `duplicates`, `notCrawled`), the cost estimate `maxCostUsd` and `expectedCostUsd`, and `combinedFileUrl`: the link to `OUTPUT.md`, all pages in one Markdown file. Very large crawls split that file into 8 MB parts (`OUTPUT-2.md`, …) listed in `combinedFileUrls`. Turn it off with **Also save one combined Markdown file** (`combinedFile: false`). ### Find specific information with a goal Looking for one thing on a site (pricing, a refund policy, API rate limits)? Add a **Goal** in a few words. Among the start URLs, sitemap pages and links found on a page, those whose link text or URL path contain these words are crawled first (a matching menu link counts as a content link), so even a small **Max pages** reaches the answer page. Each page row then has `relevantBlocks`: up to 5 sections of the page that match the goal best (cut at its headings), each with `headingPath`, `text` (up to 1,500 characters), `startIndex` (where `text` starts in the `markdown` field) and `score`; `[]` when nothing on the page matches. SUMMARY has `goalTopPages`: the 5 best-matching pages with their `score`. ```json { "urls": ["https://example.com"], "crawlScope": "domain", "maxPages": 10, "goal": "pricing plans" } ``` It is plain word matching (no AI model, English stop words ignored), so use the words the site would use. The price per page is the same, Max pages, Max link depth, URL patterns and robots.txt work as before, and without a goal the crawl and its output are unchanged. ### Use with AI agents (MCP) Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/website-markdown-crawler) to Claude, Cursor or any MCP client, then ask e.g. "Crawl the first 20 pages of docs.apify.com/academy and summarize what the course covers." Minimal input: ```json { "urls": ["https://docs.apify.com/academy"], "maxPages": 20 } ``` Failed items are not charged and carry an `errorType`. ### Use it from code Example with the Apify API: ```bash curl -X POST "https://api.apify.com/v2/acts/tidytools~website-markdown-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls":["https://docs.example.com"],"maxPages":20}' ``` #### LangChain (Python) ```python from langchain_apify import ApifyWrapper from langchain_core.documents import Document loader = ApifyWrapper().call_actor( actor_id="tidytools/website-markdown-crawler", run_input={"urls": ["https://docs.apify.com/academy"], "maxPages": 20}, dataset_mapping_function=lambda item: Document( page_content=item.get("markdown") or "", metadata={"source": item["url"], "title": item.get("title")} ), ) docs = loader.load() ``` #### LlamaIndex (Python) ```python from apify_client import ApifyClient from llama_index.core import Document client = ApifyClient("YOUR_TOKEN") run = client.actor("tidytools/website-markdown-crawler").call( run_input={"urls": ["https://docs.apify.com/academy"], "maxPages": 20}) docs = [Document(text=i["markdown"], metadata={"url": i["url"], "title": i.get("title")}) for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i.get("success")] ``` **Schedules and integrations:** run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML. ### Tips - **Documentation sites**: keep the scope on *Same folder* and keep **Remove navigation, headers and footers** on. If the start page links to nothing inside its own folder (sites that give every page its own folder, such as `/en/getting-started/`), the crawl uses the folder one level up for it (`/en/`) and lists it in SUMMARY `scopeWidened`. If the start URL redirects to another host or folder (a docs site that moved, e.g. `docs.example.io/en/stable/` to `docs.example.com/platform/stable/`), links under the redirect target are crawled as well and it is listed in SUMMARY `scopeRedirected`. If some menu text is left, add its selector to **Also remove these elements** or set the content selector to `main` or `article`. - **Calling the API?** Clean-up and tidying are on unless you send `"removeBoilerplate": false` or `"tidyMarkdown": false`. Use the plain `urls` list: the older `startUrls` field still works, but Apify rejects the whole run (HTTP 400) when it holds a bare domain or a blank line. - **Several sites at once**: set **Max pages per start URL** (e.g. 20) so each site gets pages, and group the results by `startUrl`. - **JavaScript-heavy apps** (React, Vue, Angular): Auto mode switches to the browser automatically; choose *Browser* to force it. - **Only part of a site**: use include patterns like `https://example.com/blog/**`. Start URLs are always converted, even when they do not match the patterns. - **Big sites**: raise *Max pages* and *Parallel pages*; you only pay for pages converted. Memory use is bounded: in our test a 2,700-page crawl peaked at about 83 MB, so the default 256 MB covers roughly 5,000 pages; for 10,000 pages set the run's memory to 512 MB. If a crawl would not fit, the log warns at the start, and if memory still runs short the run stops cleanly with the pages done so far (`SUMMARY.status` is `LIMIT_REACHED`, `stoppedEarly.reason` is `memory`); run again with more memory for the rest. In browser mode, pages that find no free browser ("no browser became available") are retried twice after a short wait before they count as failed (free). - **Only the pages in the sitemap**: set **Use the site's sitemap** to *Only*. - **A site blocks our servers**: under **Advanced settings**, *Auto* already retries from Apify's network; add an Apify proxy if it still fails (proxy usage is billed to your Apify account). #### What gets removed With `removeBoilerplate` on, this also removes hidden elements (`hidden` or inline `display:none`) and schema.org JSON-LD blocks before Markdown conversion. A hidden wrapper that holds most of the page (some shop themes reveal the whole page with JavaScript) is kept, and so is the `