Website to Markdown for LLM — Crawl Website to Markdown

Website to Markdown for LLM — Website Crawler for RAG

Website to Markdown is a NexGenData tool on Apify that returns page content as Markdown, text or HTML from the websites you crawl, including JavaScript-rendered pages.

Website to markdown for LLM context: point it at a docs site and get each page as clean markdown, ready to chunk for RAG.

A website to markdown crawler: start from a URL and get every page back as clean markdown or text — a website crawler for RAG and LLM pipelines.

Full-Site Scraping for AI

Public web content, for research and RAG.

📊 What you get

Clean JSON output (16 fields):

• url — Page URL
• statusCode — HTTP status of the page
• title — Page title
• description — Meta description
• author — Author, when the page states one
• publishedDate — Published date, when the page states one
• modifiedDate — Last-modified date, when the page states one
• siteName — Site name
• language — Page language
• content — Main content as plain text
• html — Cleaned HTML of the main content
• markdown — Main content as Markdown (for LLM/RAG use)
• links — Links found on the page (URL and link text)
• images — Images in the content (URL and alt text)
• wordCount — Words in the main content
• crawledAt — When the page was read (UTC)

Pricing: $0.020 per page (Pay-Per-Event) — about 50 pages per $1.

⚙️ Input

Field	Type	Default	What it does
start_urls	['array', 'string']	["https://example.com/"]	Start URLs for the website to markdown crawler. URLs to start crawling from. Accepts a list of URLs OR a single URL string OR a comma/newline-separated list. Also accepts these alternative field names: urls, startUrls, c
max_pages	integer	10	Maximum number of pages to crawl per start URL
max_depth	integer	2	Maximum crawl depth (0 = start URL only, 1 = start URL + linked pages, etc.)
same_domain	boolean	true	Only crawl pages on the same domain as the start URL
exclude_patterns	array	["/admin", "/login", "/wp-login"]	URL patterns to exclude from crawling (e.g., /admin, /login)
output_format	string	"json"	Format for extracted content
urls	['array', 'string']	""	Alternative name for start_urls. Accepts a list of URLs OR a single URL string OR comma/newline-separated.
startUrls	['array', 'string']	""	Alternative name for start_urls. Accepts a list of URLs OR a single URL string OR comma/newline-separated.
crawlUrls	['array', 'string']	""	Alternative name for start_urls. Accepts a list of URLs OR a single URL string OR comma/newline-separated.
pages	['array', 'string']	""	Alternative name for start_urls. Accepts a list of URLs OR a single URL string OR comma/newline-separated.
url	string	""	Alternative — a single URL when you only need one. Comma-separated values are auto-split.

▶️ Example input

{
  "start_urls": [
    "https://example.com/"
  ],
  "exclude_patterns": [
    "/admin",
    "/login",
    "/wp-login"
  ]
}

🧾 Example output (one real row)

{
  "url": "https://example.com/",
  "statusCode": 200,
  "title": "Example Domain",
  "description": null,
  "author": null,
  "publishedDate": null,
  "modifiedDate": null,
  "siteName": null,
  "language": "en",
  "content": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
  "html": "<body><p>This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.</p></body>",
  "markdown": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
  "links": [],
  "images": [],
  "wordCount": 25,
  "crawledAt": "2026-11-03T20:00:30.692216Z"
}

🤖 Use with AI agents

Point Claude, the OpenAI Agents SDK, an n8n flow or any MCP-aware client at it.

Sample agent prompt:

Run Website Crawler on my input and return the structured results.

Agentic payments (x402): Supports agentic payment via x402 — call this actor with USDC, no API key required.

🔗 Related tools — Web crawling for AI/RAG

RAG Web Browser

---

A NexGenData utility actor.