Website to Markdown & RAG Chunks for LLMs

Formerly AI-Ready Webpage Extractor. Same Actor, same API name (s3nafps/ai-ready-webpage-extractor).

Website to Markdown & RAG Chunks converts public web pages, documentation sites and blogs into clean, heading-preserving Markdown — and optional heading-aware RAG chunks — ready for LLMs, AI agents and vector databases. It strips menus, cookie banners, footers and ads, fetches HTML first and opens a real browser only when the page comes back empty, and charges only for successfully extracted pages.

What it does

For each page it returns:

• Markdown (or plain text) with headings, lists, code blocks (with language), tables and links preserved
• Optional RAG chunks with heading path, token estimate and overlap — ready to embed
• Metadata when present: title, description, author, published/modified dates, canonical URL, language, schema.org JSON-LD
• Optional links (anchor text) and tables as row arrays
• One dataset row per page, plus a run OUTPUT summary of failed pages and reasons

Crawl modes:

Mode	crawlMode	Behavior
Only these URLs	none (default)	Extract exactly the URLs you list
Follow links on the same site	sameDomain	Also extract linked pages up to maxCrawlDepth
Use the site's sitemap	sitemap	Discover pages via robots.txt / sitemap.xml (falls back to link-following if none match)

Who it’s for

• Builders wiring RAG pipelines, custom GPTs or support chatbots on public docs
• Agent developers who need a reliable “read this page as Markdown” tool
• Researchers collecting articles with author/date metadata
• Anyone who wants predictable per-page pricing instead of open-ended platform-usage crawls

Use cases

• Feed a help center or docs site into a RAG index (Pinecone, Qdrant, Weaviate, pgvector, …)
• Give Claude, ChatGPT, Cursor or another MCP client a web-reading tool that returns compact Markdown
• Build a searchable knowledge base from public documentation
• Extract pricing or product/spec tables as structured rows alongside Markdown
• Batch-convert blog posts for LLM fine-tuning or summarization workflows

Quick start (first run)

1. Leave the prefilled URL (https://docs.apify.com/actors) or paste your own.
2. Keep Crawl mode = Only these URLs, Maximum pages = 5.
3. Optionally turn on Split content into RAG chunks.
4. Click Start → open the dataset (Overview or RAG chunks views) or export JSON/CSV.

Input example

{
  "urls": ["https://docs.apify.com"],
  "crawlMode": "sitemap",
  "urlMustContain": "docs.apify.com/actors/",
  "maxPages": 50,
  "chunkContent": true,
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 100,
  "removeBoilerplate": true,
  "renderJavaScript": "auto",
  "outputFormat": "markdown"
}

Key fields: urls (required), crawlMode (none / sameDomain / sitemap), maxPages, maxCrawlDepth, urlMustContain, outputFormat (markdown / text / both), chunkContent, chunkSizeTokens, chunkOverlapTokens, removeBoilerplate, includeLinks, includeTables, includeMetadata, renderJavaScript (auto / never / always), includeFailedPages, maxConcurrency.

Output example (shortened)

{
  "sourceUrl": "https://docs.apify.com/actors",
  "httpStatus": 200,
  "title": "Actors | Platform | Apify Documentation",
  "canonicalUrl": "https://docs.apify.com/actors",
  "language": "en",
  "mainContentMarkdown": "# Actors\n\nActors are serverless cloud programs...",
  "chunks": [
    {
      "index": 0,
      "headingPath": ["Actors"],
      "estimatedTokens": 412
    }
  ],
  "wordCount": 480,
  "estimatedTokens": 669,
  "renderedWithBrowser": false,
  "extractionStatus": "succeeded"
}

Dataset views: Overview (one row per page) and RAG chunks (one row per chunk).

Pricing (effective 2026-11-23)

Pay per event. Failed pages are not charged. JS rendering, chunking, crawling, tables, metadata and JSON-LD are included in the page price. Default run memory is 1024 MB → one Actor Start event.

Event	FREE	BRONZE	SILVER	GOLD+
Actor Start	$0.05	$0.05	$0.05	$0.05
Page (success only)	$0.02	$0.018	$0.016	$0.014

Cost per 1,000 successful pages (plus one start fee):

Plan tier	Pages	Start	Total ≈
FREE	1,000 × $0.02 = $20.00	$0.05	$20.05
BRONZE	1,000 × $0.018 = $18.00	$0.05	$18.05
SILVER	1,000 × $0.016 = $16.00	$0.05	$16.05
GOLD / PLATINUM / DIAMOND	1,000 × $0.014 = $14.00	$0.05	$14.05

Examples: 1 page on FREE ≈ $0.07; 100-page docs section on FREE ≈ $2.05. Set Maximum cost per run in the run options to cap spend.

Until 2026-11-23 the Store still bills the older rates (Start $0.00005, Page from $0.003). Update this README the day the new prices activate if you publish earlier.

How it differs from a general web scraper

Built for LLM input, not raw HTML dumps: main-content extraction, structure-preserving Markdown, optional heading-aware chunks with token estimates, and metadata. You get text you can embed or pass to a model as-is.

Compared with heavyweight site crawlers billed on platform usage, this Actor is PPE and page-scoped — useful when you want a predictable bill for “N docs pages → Markdown/chunks.”

Related Actors: PDF & Word to Markdown for RAG (same chunk idea, with page numbers).

Integrations

• Apify API / JS & Python clients — client.actor("s3nafps/ai-ready-webpage-extractor").call(input)
• MCP (Claude, Cursor, VS Code, …) via Apify’s hosted MCP:
  https://mcp.apify.com/?actors=s3nafps/ai-ready-webpage-extractor
  (or tools list including this Actor — see mcp.apify.com)
• n8n / Make / Zapier — run Actor → read dataset items
• LangChain / LlamaIndex — load the run dataset (e.g. LangChain ApifyDatasetLoader) and embed chunks or mainContentMarkdown

FAQ

What’s the best way to convert a docs site to Markdown for ChatGPT or Claude?
Use crawlMode: "sitemap" with a urlMustContain filter (e.g. /docs/), or list URLs. Pass mainContentMarkdown to the model, or connect the Actor through Apify MCP so the assistant calls it.

Can I turn a whole documentation site into RAG chunks?
Yes. Sitemap (or same-site) crawl + chunkContent: true. Each chunk includes a heading path and token estimate. Use the RAG chunks dataset view.

Does it work on React / Next.js / Vue sites?
Yes. With renderJavaScript: "auto" (default), pages that return empty HTML are opened in a real browser.

How much will 1,000 pages cost after Nov 23?
About $20.05 on the FREE tier, $14.05 on Gold+, including one Actor Start. Failed pages are free.

Which vector databases work with the output?
Any that accept text/embeddings — e.g. Pinecone, Qdrant, Weaviate, pgvector — via the chunks view or your own loader.

Can it scrape pages behind a login or paywall?
No. Public pages only. Blocked or login pages are reported as failed and not charged.

Is it legal to convert websites to Markdown?
It reads publicly available pages only. Respect each site’s terms of use and applicable law.

Limitations

Public HTML pages only; non-HTML files are skipped; main-content detection is heuristic; token counts are estimates (~4 characters/token).

About the author

Built and maintained by Mohamed Senator — all Actors: apify.com/s3nafps.