# Website Content Crawler to Markdown for RAG & LLMs (`rod_analytics/website-to-markdown`) Actor Low-cost website content crawler: turn any site into clean Markdown for RAG, LLMs and AI agents at a flat $0.30 per 1,000 pages. Main-content extraction, GFM tables, RAG chunks, llms.txt per domain, robots.txt respected. - **URL**: https://apify.com/rod_analytics/website-to-markdown.md - **Developed by:** [Rod Services](https://apify.com/rod_analytics) (community) - **Categories:** AI, Developer tools - **Features:** Limited permissions - **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.21 / 1,000 page crawleds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README **Website content crawler to Markdown for RAG and LLMs: crawl any site into clean Markdown, RAG chunks and llms.txt.** A low-cost Website Content Crawler alternative: flat **$0.30 per 1,000 pages**, fast HTTP crawling, no compute bill to estimate. ### What you get For every page of the site: - **Clean Markdown** of the main content: no menus, sidebars, footers or cookie banners. GitHub-flavoured tables, code blocks and absolute links. - Optional **RAG chunks** split at headings, paragraphs and code blocks, with token estimates and the heading path. - **Metadata**: title, description, H1, language, last modified date, canonical URL and a content hash for change detection. - An **llms.txt** index and an **llms-full.txt** file per domain in the key-value store. It is a **fast HTTP crawler with no browser**, so a 200 page documentation site takes about a minute. Pages that are only an empty JavaScript shell can be rendered in headless Chrome on request. Duplicates, robots.txt blocks, errors and empty pages are never charged. Typical uses: - **Knowledge base for a chatbot or AI agent.** Crawl docs, a help center or a blog and load the chunks into a vector database. - **LLM fine-tuning data.** Clean Markdown, one record per page, with language and word count. - **llms.txt for your own site.** A ready index and a full text file that AI assistants can read. - **Keep an index fresh.** Schedule weekly runs. `contentHash` and `lastModified` show which pages changed. ### Why pick this over a browser-based crawler | | This Actor | Apify Website Content Crawler | | ---------------- | ---------------------------------------------------------------------- | ------------------------------------------------------- | | Pricing | Flat $0.30 per 1,000 HTTP pages, $6 per 1,000 browser pages | Free Actor, you pay platform compute for the run | | Default fetching | Plain HTTP, browser only as an opt-in fallback | Adaptive, can use a headless browser | | Cost to plan | Pages x price, capped by **Max pages** | Depends on memory, browser use and run time | | Best for | Docs, blogs, help centers, marketing sites, most server-rendered sites | Heavy JavaScript apps and extras such as file downloads | Most documentation and content sites, including Docusaurus, GitBook, WordPress, MkDocs and Next.js sites, ship HTML and work over plain HTTP. For those, a flat price per page is the cheapest way to get RAG data. ### Quick start 1. Open the **Input** tab and paste one or more start URLs, for example `https://docs.example.com/`. 2. Set **Max pages**. This is also your cost cap. 3. Optional: turn on **Split into RAG chunks** and pick a chunk size. 4. Click **Start**. The prefilled example saves 10 pages of Apify Academy in under 30 seconds. 5. Download the dataset as JSON, CSV, Excel or HTML, or open the llms.txt files in the Storage tab. **Single page mode:** set **Max crawl depth** to `0`. Only the start URLs are fetched, which turns the Actor into a URL to Markdown converter for a list of pages. By default the crawler stays under the start URL path: `https://example.com/docs` crawls `/docs` and everything below it. Redirects are followed, and the crawl stays under the new path. From code, one HTTP call runs the crawl and returns the pages: ```bash curl -X POST "https://api.apify.com/v2/acts/rod_analytics~website-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"startUrls": [{"url": "https://docs.apify.com/academy"}], "maxPages": 50, "includeChunks": true}' ``` Load the chunks into LangChain documents: ```python from apify_client import ApifyClient from langchain_core.documents import Document client = ApifyClient("APIFY_TOKEN") run = client.actor("rod_analytics/website-to-markdown").call( run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 500, "includeChunks": True} ) docs = [ Document(page_content=c["text"], metadata={"source": page["url"], "title": page["title"], "headings": c["headings"]}) for page in client.dataset(run["defaultDatasetId"]).iterate_items() for c in page.get("chunks") or [] ] ``` ### Input example ```json { "startUrls": [{ "url": "https://docs.apify.com/academy" }], "maxPages": 10, "includeChunks": true, "chunkSize": 500, "chunkOverlap": 50 } ``` | Field | What it does | Default | | --------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------ | | `startUrls` | Where the crawl starts | | | `maxPages` | Stop after this many saved pages | 100 | | `maxCrawlDepth` | Links away from a start URL. `0` fetches only the start URLs | 20 | | `includeGlobs` / `excludeGlobs` | URL patterns, `**` matches any path. `.../blog/**` also matches `.../blog` | none | | `sameDomainOnly` / `stayWithinStartPath` | Crawl scope | on / on | | `useSitemaps` | Also queue URLs from sitemap.xml | off | | `respectRobotsTxt` | Skip pages disallowed by robots.txt | on | | `extractionMode` | `readability`, `main` or `full` page | readability | | `removeSelectors` | Extra CSS selectors to drop | none | | `includeChunks`, `chunkSize`, `chunkOverlap` | RAG chunks, sizes in estimated tokens | off, 500, 50 | | `dedupeContent` | Skip same canonical URL or identical content | on | | `jsRenderingFallback` | Render near-empty pages in headless Chrome | off | | `maxConcurrency`, `maxConcurrencyPerDomain`, `delayBetweenRequestsMs` | Politeness per domain | 10, 5, 0 | ### Output sample One dataset item per page, shortened. The Output tab has four views: Overview, Markdown, Metadata and RAG chunks. ```json { "url": "https://docs.apify.com/academy/api-scraping", "canonicalUrl": "https://docs.apify.com/academy/api-scraping", "title": "API scraping | Academy | Apify Documentation", "description": "Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.", "language": "en", "h1": "API scraping", "markdown": "# API scraping\n\nAPI scraping is locating a website's API endpoints, and fetching the desired data directly from their API...", "wordCount": 868, "chunkCount": 6, "chunks": [ { "index": 0, "headings": "API scraping", "text": "# API scraping\n\nAPI scraping is locating a website's API endpoints...", "charCount": 1790, "tokenEstimate": 448 } ], "linksCount": 73, "depth": 1, "statusCode": 200, "fetchMode": "http", "contentHash": "3f1c0a9e5b7d2c4e8a61", "crawledAt": "2026-09-27T11:32:57.120Z", "warning": null } ``` The key-value store also holds: - `llms-.txt`: an [llms.txt](https://llmstxt.org) index with the title, URL and description of every page. - `llms-full-.txt`: all Markdown of the domain in one file, each page wrapped in ``. Split into parts above 8 MB. - `RUN-SUMMARY`: saved, duplicate, blocked and failed counts, plus the first 500 problem URLs. ### Pricing Pay per event, no compute charges. | Event | Free tier | Bronze | Silver | Gold | | ------------------------------- | --------- | -------- | -------- | -------- | | Page crawled over HTTP | $0.0003 | $0.00027 | $0.00024 | $0.00021 | | Page rendered in Chrome, opt-in | $0.006 | $0.0054 | $0.0048 | $0.0042 | | Run start, per GB of memory | $0.001 | $0.001 | $0.001 | $0.001 | - 1,000 pages cost $0.30 on the free tier and $0.21 on Gold. 10,000 pages cost $3. - Duplicates, robots.txt blocks, errors and empty pages are **not charged**. - The Apify free plan's $5 monthly credit covers about 16,000 pages. - Your discount tier follows your Apify subscription plan. - Set **Maximum cost per run** in Run options and the crawler stops exactly at that budget. ### FAQ **Does it work on JavaScript sites?** Most sites ship HTML and work over plain HTTP. Pure single page apps return an empty shell. Enable **JavaScript rendering fallback** for them, at the browser page price. **Does it respect robots.txt?** Yes, by default. Disallowed pages are skipped and listed in `RUN-SUMMARY`. You can turn it off only at your own responsibility, and the run then logs a warning. **How are duplicates detected?** By final URL, by `rel=canonical` and by a hash of the Markdown. **Which proxies can I use?** No proxy, which is the default, Apify datacenter proxy, or your own proxy URLs. Residential and SERP proxies are not supported. A run that asks for them stops at the start with a clear message. **What if nothing is saved?** The run still ends as succeeded. The dataset holds one free row with `errorCode` and `howToFix`, for example for an empty input, a 404 start page or a JavaScript-only site. **Tips.** Use include globs such as `https://example.com/docs/**` to skip blogs and tags. Turn on sitemaps to find pages not linked from the menu. Keep 1024 MB memory for HTTP runs and 2048 MB with the JavaScript fallback. 300 to 800 token chunks with 10% overlap suit most embedding models. **Known limits.** No login or forms. PDF and other files are skipped. Very large pages are cut at 5 MB of HTML and 3.5 million characters of Markdown, with a `warning`. Hash-routed single page apps (`/#/page`) show as one URL. Token counts are estimates, about 4 characters per token. **Is crawling legal?** Crawling public pages is generally allowed, but you are responsible for each site's terms, copyright and privacy law. Do not crawl personal data without a legal basis. Found a bug or need an export to your vector database? Open an issue in the **Issues** tab. ### Related Actors | Actor | What it does | | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | [PDF to Markdown & Text Extractor with OCR](https://apify.com/rod_analytics/doc-to-markdown) | Turn PDF, DOCX and scans into Markdown or text | | [SEO Audit Tool: Site Crawler, Broken Links & Score](https://apify.com/rod_analytics/seo-site-audit) | Crawl a site for broken links and SEO issues | | [PageSpeed Insights & Lighthouse Bulk Checker](https://apify.com/rod_analytics/lighthouse-audit) | Core Web Vitals and Lighthouse scores for many URLs | | [Tech Stack Detector: Wappalyzer & BuiltWith Alternative](https://apify.com/rod_analytics/tech-stack-detector) | See which CMS, analytics and frameworks a site uses | | [Bulk Domain Lookup: DNS, WHOIS, SPF, DMARC & SSL](https://apify.com/rod_analytics/domain-security-audit) | DNS, mail security and SSL checks for domain lists | | [Speech to Text: Audio, Video & Podcast Transcriber](https://apify.com/rod_analytics/media-transcriber) | Transcribe audio, video and podcasts | | [Email Validator & Verifier: Bulk MX & Disposable Check](https://apify.com/rod_analytics/bulk-email-verifier) | Clean email lists before you send | | [Website Contact Scraper: Emails, Phones & Social Links](https://apify.com/rod_analytics/company-contact-extractor) | Emails, phones and social links from company websites | | [VIES VAT Checker & EU Company Registry Lookup](https://apify.com/rod_analytics/eu-company-lookup) | Validate EU VAT numbers and look up companies | | [Companies House Scraper: UK Company Search & Officers](https://apify.com/rod_analytics/uk-companies-house) | UK company profiles, officers and filings | | [EU TED Tenders Scraper & Alerts: Public Procurement](https://apify.com/rod_analytics/ted-tenders) | EU public tenders with daily alerts | | [Greenhouse, Lever & Ashby Job Scraper: 7 ATS in One Feed](https://apify.com/rod_analytics/multi-ats-jobs-api) | Jobs from 7 ATS platforms in one feed | | [GitHub Scraper: Repo Stats, Stars, Trending & Search](https://apify.com/rod_analytics/github-repo-stats) | Stars, activity and trending GitHub repos | # Changelog This Actor's version history is a separate document: https://apify.com/rod_analytics/website-to-markdown/changelog.md # Actor input Schema ## `startUrls` (type: `array`): Pages where the crawl starts. By default only pages under the same path are crawled: https://example.com/docs crawls /docs and everything below it. ## `maxPages` (type: `integer`): Stop after this many pages are saved. You pay per saved page, so this is also your cost cap. ## `maxCrawlDepth` (type: `integer`): How many links away from a start URL the crawler may go. 0 crawls only the start URLs (plus sitemap URLs, if enabled). ## `includeGlobs` (type: `array`): Only crawl URLs matching at least one pattern. Use \*\* for any path, for example https://example.com/blog/\*\*. When set, this replaces the "Stay within start path" rule. ## `excludeGlobs` (type: `array`): Never crawl URLs matching any of these patterns, for example **/tag/** or https://example.com/login\*\*. ## `sameDomainOnly` (type: `boolean`): Follow links only on the start URL hosts. www.example.com and example.com count as the same host. ## `stayWithinStartPath` (type: `boolean`): Only follow links under the start URL path. Turn off to crawl the whole site. ## `useSitemaps` (type: `boolean`): Find sitemaps via robots.txt and /sitemap.xml and queue the URLs they list (still filtered by the rules above). Finds pages that are not linked from anywhere. ## `respectRobotsTxt` (type: `boolean`): WARNING: keep this on. Pages disallowed by the site robots.txt are skipped and never charged. Turning it off logs a warning, and you alone are responsible for having permission to crawl the site. ## `extractionMode` (type: `string`): Readability: finds the main article like Firefox Reader View and drops menus, sidebars, footers and cookie banners (best for RAG). Main element: only removes known boilerplate and keeps the main or article element. Full page: keeps everything except scripts and cookie banners. ## `removeSelectors` (type: `string`): Extra elements to drop before conversion, as one CSS selector list. Example: .newsletter, #comments, .related-posts ## `dedupeContent` (type: `boolean`): Skip pages whose canonical URL was already saved or whose Markdown is identical to a saved page. Duplicates are not charged. ## `saveLlmsTxt` (type: `boolean`): Write llms-.txt (index of pages) and llms-full-.txt (all Markdown in one file) to the key-value store at the end of the run. ## `includeChunks` (type: `boolean`): Add a chunks array to each page, split at headings, paragraphs and code blocks, ready for embeddings and a vector database. ## `chunkSize` (type: `integer`): Target size of one chunk in estimated tokens (about 4 characters per token). ## `chunkOverlap` (type: `integer`): How many tokens of the previous chunk are repeated at the start of the next one. Capped at half the chunk size. ## `jsRenderingFallback` (type: `boolean`): Off by default. When a page has almost no text over HTTP (a JavaScript app shell), open it once in headless Chrome and extract the rendered content. Rendered pages are billed as a separate, more expensive event. Use at least 2048 MB memory. ## `minWordsForHttp` (type: `integer`): With the fallback on, pages with fewer words than this over HTTP are rendered in the browser. ## `maxConcurrency` (type: `integer`): Maximum parallel requests in total. ## `maxConcurrencyPerDomain` (type: `integer`): Maximum parallel requests to one host. Lower it for small sites. ## `delayBetweenRequestsMs` (type: `integer`): Minimum time between two request starts to the same host. 0 means no delay. ## `requestTimeoutSecs` (type: `integer`): Give up on a page after this many seconds. ## `maxRequestRetries` (type: `integer`): Retries for network errors and blocked responses. Failed pages are not charged. ## `proxyConfiguration` (type: `object`): Optional. Most sites work without a proxy. Use Apify datacenter proxy or your own proxy URLs if a site blocks you. Residential and SERP proxies are not supported. ## Actor input object example ```json { "startUrls": [ { "url": "https://docs.apify.com/academy" } ], "maxPages": 10, "maxCrawlDepth": 20, "includeGlobs": [], "excludeGlobs": [], "sameDomainOnly": true, "stayWithinStartPath": true, "useSitemaps": false, "respectRobotsTxt": true, "extractionMode": "readability", "dedupeContent": true, "saveLlmsTxt": true, "includeChunks": false, "chunkSize": 500, "chunkOverlap": 50, "jsRenderingFallback": false, "minWordsForHttp": 30, "maxConcurrency": 10, "maxConcurrencyPerDomain": 5, "delayBetweenRequestsMs": 0, "requestTimeoutSecs": 30, "maxRequestRetries": 2, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `pages` (type: `string`): No description ## `markdown` (type: `string`): No description ## `files` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://docs.apify.com/academy" } ], "maxPages": 10 }; // Run the Actor and wait for it to finish const run = await client.actor("rod_analytics/website-to-markdown").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy" }], "maxPages": 10, } # Run the Actor and wait for it to finish run = client.actor("rod_analytics/website-to-markdown").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://docs.apify.com/academy" } ], "maxPages": 10 }' | apify call rod_analytics/website-to-markdown --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,rod_analytics/website-to-markdown" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/ZbhYRQVxEGyfcxxp7/builds/zWFG3I0cBwydP9s9Y/openapi.json