# Website to Markdown — Content Crawler for LLM & RAG (`webdatatools/website-to-markdown`) Actor

Convert web pages into clean Markdown for LLMs, RAG and vector databases. Pay per delivered result: $1/1,000 on Free; from $0.60/1,000 on Diamond. $0.00005 per run start.

- **URL**: https://apify.com/webdatatools/website-to-markdown.md
- **Developed by:** [Murat Uzun](https://apify.com/webdatatools) (community)
- **Categories:** AI, Developer tools, Agents
- **Features:** Limited permissions
- **Stats:** 7 total users, 3 monthly users, 99.2% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Website to Markdown Crawler?

Website to Markdown Crawler is an Apify Actor that turns any website into clean, LLM-ready **Markdown** — one row per page, with navigation, footers, sidebars and cookie banners stripped, headings and code blocks preserved, and every link made absolute. It runs on plain HTTP with [Crawlee](https://crawlee.dev)'s `CheerioCrawler` — **no headless browser** — which is what makes it fast and roughly **10× cheaper** than browser-based content crawlers. Feed it a start URL (or a whole `/sitemap.xml`) and it does a breadth-first crawl up to your page budget, ready to feed an **LLM, a RAG pipeline, LangChain/LlamaIndex loader, or a vector DB** (Pinecone, Qdrant, Weaviate, pgvector) in minutes.

**Pricing:** Website to Markdown costs **$1.00 per 1,000 results** (pay per result, no subscription; Apify's free plan credit covers small runs).

### Why use Website to Markdown Crawler?

- **Feed an LLM or RAG pipeline in minutes.** Point it at your docs site, a competitor's help center, or your own product site and get clean Markdown ready for chunking and embedding — no HTML soup to clean up first.
- **10× cheaper, no browser.** `apify/website-content-crawler` and similar Actors spin up a browser per page. This Actor fetches raw HTML over HTTP, so it's faster and priced at **$1 per 1,000 pages**.
- **Predictable cost.** `maxPages` is the billed unit and the crawl stops at exactly that count — no surprise runs.
- **Sitemap-aware BFS crawl.** Optionally seeds from `/sitemap.xml` (including nested sitemap indexes) so you reach real content pages immediately instead of waiting for link discovery.

### How to use Website to Markdown Crawler

1. Paste one or more pages into **Start URLs**, e.g. `https://docs.apify.com/platform` or your own docs/blog root.
2. Set **Max pages** — this is the billed unit ($1 per 1,000 pages) and the crawl stops exactly there.
3. Optionally set **Include path prefixes** (e.g. `/docs`) to stay inside one section of a larger site.
4. Choose **Output format**: Markdown, plain text, or both.
5. Click **Start**, then download the dataset as JSON, CSV, Excel or HTML, or pull it straight into your RAG pipeline via the API.

### Example input

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "maxPages": 50,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "useSitemap": true,
  "outputFormat": "markdown"
}
```

### Example output

```json
{
  "url": "https://crawlee.dev/docs",
  "finalUrl": "https://crawlee.dev/js/docs/quick-start",
  "statusCode": 200,
  "title": "Quick Start | Crawlee for JavaScript",
  "description": "Crawlee helps you build reliable scrapers. Fast.",
  "lang": "en",
  "canonical": "https://crawlee.dev/js/docs/quick-start",
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two...\n\n## Choose your crawler\n\n### CheerioCrawler\n\nThis is a plain HTTP crawler...",
  "text": null,
  "wordCount": 1484,
  "headings": ["Quick Start", "Choose your crawler", "CheerioCrawler", "PuppeteerCrawler"],
  "links": ["https://crawlee.dev/js/docs/introduction", "https://crawlee.dev/js/docs/guides"],
  "depth": 0,
  "contentHash": "3f9a1c2e...",
  "crawledAt": "2026-09-12T18:21:07.000Z",
  "error": null
}
```

### What data does Website to Markdown Crawler extract?

One row per crawled page:

| Field | Type | Description |
|---|---|---|
| `url` / `finalUrl` | string | Requested URL and URL after redirects |
| `statusCode` | integer | HTTP status code, `null` on a failed request |
| `title` / `description` | string | Page `<title>` (or first `<h1>`) and meta description |
| `lang` | string | `<html lang>` attribute |
| `canonical` | string | `<link rel=canonical>`, resolved to an absolute URL |
| `markdown` | string | Main content as Markdown — headings, lists, code blocks, tables, absolute links/images |
| `text` | string | Main content as plain text |
| `wordCount` | integer | Word count of the extracted content |
| `headings` | array | H1-H3 text, in order, up to 50 |
| `links` | array | Same-domain absolute links found on the page, up to 200 |
| `depth` | integer | Link depth from the nearest start URL |
| `contentHash` | string | SHA-1 of the plain text — diff two runs to detect changed pages |
| `crawledAt` | string | ISO timestamp |
| `error` | string | Always `null`: pages that fail are not charged and are listed in the run's `ERRORS` record instead |
| `chunks` | array | With **Add RAG chunks**: `{ text, tokens, headingPath }` per chunk, otherwise `null` |
| `changeStatus` | string | With **Detect changes**: `new`, `changed` or `unchanged`, otherwise `null` |

`markdown`/`text` are populated according to the **Output format** input (`markdown`, `text`, or `both`); the unused one is `null`.

### Input parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | string | none | Fast single-page mode for AI agents: fetch only this page (no sitemap, no link following); overrides `startUrls` and `maxPages` |
| `startUrls` | array | `docs.apify.com/platform` | Page(s) to start crawling from |
| `maxPages` | integer | 50 | Billed unit; the crawl stops at exactly this count (1-5000) |
| `maxDepth` | integer | 3 | Link depth from a start URL (0-10) |
| `sameDomainOnly` | boolean | true | Only follow links on any supplied start domain (`www.` counts as the same) |
| `includePathPrefixes` | array | none | Only crawl paths starting with one of these, e.g. `/docs` |
| `excludePathPatterns` | array | images/PDF/CSS/JS, `/login`, `/signup`, `?replytocom=` | Regexes tested against the full URL; a match is skipped |
| `useSitemap` | boolean | true | Also seed from `/sitemap.xml` (follows one nested sitemap-index level) |
| `outputFormat` | string | `markdown` | `markdown`, `text`, or `both` |
| `removeSelectors` | array | none | Extra CSS selectors to strip before extraction |
| `maxConcurrency` | integer | 10 | Parallel requests (1-50) |
| `maxPagesPerDomain` | integer | none | Cap pages crawled from any one domain (`www.` ignored) |
| `maxConsecutiveFailures` | integer | 20 | Stop the crawl after this many failed pages in a row; `0` = never |
| `respectRobots` | boolean | true | Skip URLs disallowed by `robots.txt` |
| `proxyConfiguration` | object | none | Optional Apify Proxy configuration |
| `chunkForRag` | boolean | false | Add a `chunks` array (heading-aware, exact o200k token counts) to every page |
| `chunkSize` | integer | 500 | Max tokens per chunk (50-8000) |
| `changeDetection` | boolean | false | Add `changeStatus` (new / changed / unchanged) versus the previous run with the same start URLs |
| `skipUnchangedContent` | boolean | true | With change detection on, unchanged pages get empty markdown/text/chunks |

### Pricing

Website to Markdown Crawler uses pay-per-event pricing: **$1 per 1,000 pages** ($0.001 per page), plus a negligible actor-start fee. There is no separate compute-unit or browser billing because no browser is used. Set **Max pages** (and, on the platform, **Maximum cost per run**) to cap spend — the Actor trims its own crawl to stay within budget and never bills more pages than it delivers.

### Website to Markdown Crawler vs. browser-based content crawlers

Browser-based crawlers such as `apify/website-content-crawler` load every page in a real (often headless) browser before extracting content — that's necessary for JavaScript-rendered single-page apps, but it's slow and expensive for the huge majority of documentation sites, blogs and marketing pages that are plain server-rendered HTML. This Actor skips the browser entirely: a page that takes a browser-based crawler several seconds and a full render cycle is a sub-second HTTP fetch here, at a fraction of the price. The trade-off is explicit: **sites that only render content client-side with JavaScript (pure SPAs) will come back with little or no text** — see Limitations below.

### RAG-ready chunks in the same run

Turn on **Add RAG chunks** and every page row gets a `chunks` array: heading-aware pieces of the Markdown, each with its heading path and an exact GPT (o200k) token count — embed them straight into Pinecone, Qdrant, pgvector or any vector DB. Code blocks stay whole and every chunk is at most **Chunk size** tokens. No extra charge: chunks are part of the page result.

### Keep a knowledge base fresh: change detection

Turn on **Detect changes since last run** and schedule the Actor. Each page gets `changeStatus` — `new`, `changed` or `unchanged` — compared with the previous run that used the same start URLs (the state is kept in a key-value store in your own account). With **Leave content empty for unchanged pages** on, unchanged rows carry no Markdown, so you only re-embed what changed. Every crawled page still counts as one result.

### Using Website to Markdown Crawler with AI agents, LangChain and LlamaIndex

Website to Markdown Crawler runs on pay-per-event pricing with limited permissions, so it's callable through the [Apify MCP server](https://mcp.apify.com) directly from an AI agent — pass `startUrls` and `maxPages` and get back one Markdown row per page.

**For AI agents: pass `url`.** `{"url": "https://crawlee.dev/js/docs/quick-start"}` fetches just that one page — no sitemap, no link following, one row, one result billed — so the answer is ready inside the MCP `call-actor` wait (at most 45 s). In a cloud test on 9 October 2026 that page (1,484 words) came back as 17,649 characters of Markdown and the run took about 4 seconds from container start to finish. `url` overrides `startUrls` and `maxPages`; use `startUrls` to crawl a site. If the page cannot be fetched, the run's status message says why and nothing is billed for it. The dataset also drops straight into a RAG pipeline: use Apify's [LangChain](https://python.langchain.com/docs/integrations/document_loaders/apify_dataset/) or [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/data_connectors/ApifyDatasetReader/) `ApifyDatasetLoader` to turn the dataset into `Document` objects, then chunk and embed into Pinecone, Qdrant, Weaviate or pgvector. It also connects through n8n, Make and Zapier via Apify's standard integrations.

**Timeouts and delivery state.** Confirmed pages are saved as they are converted. The crawler stops taking new pages near the deadline. Read the `OUTPUT` record for `resultRows`, `failedPages`, `partial`, `stoppedForTime`, `chargeLimitReached`, and `deliveryUnconfirmed`; counters cover this process attempt. If a dataset write loses its acknowledgement or is still in flight at the hard stop, the run fails with unconfirmed delivery. The application does not repeat that write; inspect the dataset before manually retrying. This is not an exactly-once guarantee across SDK retries, migrations, or new runs.

**Sizing a run.** A typical site crawls at 2–10 pages per second (in a 60-second local test, 282 docs.apify.com pages at `maxConcurrency` 3), after a few seconds of start-up and sitemap reading: `maxPages` 100 fits a 60-second `timeout` on most sites; for thousands of pages give the run several minutes. `maxTotalChargeUsd` caps the cost — the Actor lowers `maxPages` to what the budget covers ($1 per 1,000 pages, so $0.05 = 50 pages) and says so in the status message.

**Source failures and partial runs.** The crawler stops after `maxConsecutiveFailures` consecutive failed pages. If every attempted source fails, the run is FAILED with zero result rows and no dataset-result events. Successful partial output remains available with `OUTPUT.partial=true`; failed sources are in `ERRORS`. Link-discovery errors after a confirmed write are recorded separately in `DISCOVERY_ERRORS` and do not repeat the delivered row. Startup fees and platform usage may still apply. A zero-result charging allowance is reported as `NO_RESULT_BUDGET`, not a source failure. Valid empty HTML and change-detection rows retain their existing output behavior.

### Limitations

- **JavaScript-rendered single-page apps are the main limitation.** This Actor fetches raw HTML over HTTP and never executes page JavaScript, so a site that renders its content entirely client-side (no content in the initial HTML) will come back with an empty or near-empty row. It does follow `<meta http-equiv="refresh">` redirect stubs (one hop) and ordinary HTTP redirects, which covers a common class of "empty landing page" issue, but not full client-side rendering. If you hit this, look for a browser-based crawler instead.
- Interactive widgets that swap content via JavaScript (tabs, accordions with lazy content) are captured in whatever state they render in the raw HTML — usually just the first/default tab.
- **Short run timeouts end cleanly.** If the run timeout is reached before `maxPages`, the crawler stops shortly before the deadline and the run finishes as SUCCEEDED with every page crawled so far (you pay only for those). Raise the timeout or lower `maxPages` for a full crawl.
- The main-content heuristic (no `@mozilla/readability`) is tuned for documentation and blog layouts (`article`, `main`, `#content`, `.content`); unusual layouts may need `removeSelectors` to clean up.

### Need bulk volume or a custom scraper?

Need a fixed monthly price, a large recurring run, extra output fields, or a scraper for a site this Actor does not cover? Open an issue on this Actor's **Issues** tab and include: the target site or URL pattern, the expected volume per run and how often you run it, and the output fields you need. We reply there.

### FAQ

**Is this legal?** The Actor reads only public HTML. You're responsible for crawling sites you're allowed to and respecting their terms of service.

**Why is a row empty?** Almost always a JavaScript-rendered page — see Limitations above. Check the `statusCode` field (failed pages are in the `ERRORS` record); a `200` with an empty `markdown` usually means client-side rendering.

**How do I stay inside one section of a site?** Set `includePathPrefixes`, e.g. `["/docs"]`, and/or turn off `useSitemap` if the sitemap covers more than you want.

**How do I control cost?** Set `maxPages`; the Actor never crawls or bills past it. On the platform, also set **Maximum cost per run**. `maxPagesPerDomain` caps any one domain, and `maxConsecutiveFailures` (default 20) stops a run whose pages keep failing.

**How do I get one page fast from an AI agent?** Send `{"url": "https://…"}` — single-page mode skips the sitemap and link following and returns one row in seconds.

**Can I detect changed pages between runs?** Yes — compare `contentHash` (SHA-1 of the extracted text) across runs to see which pages changed.

**Does this work with n8n, Make or Zapier?** Yes, through Apify's standard integrations, and through LangChain/LlamaIndex document loaders for RAG pipelines.

### Related Actors

Part of the **webdatatools** web-intelligence suite — every Actor is pay-per-event, reads public data
without a login, and returns one clean row per entity:

Browse the whole suite at [webdatatools](https://paulet4a-commits.github.io/webdatatools/), or call ten of
these Actors straight from Claude, Cursor or Cline with the
[webdatatools MCP server](https://github.com/paulet4a-commits/webdatatools-mcp-server).

**Website & domain intelligence**

- [Email Extractor — Website Contact & Social Finder](https://apify.com/webdatatools/contact-extractor) — e-mails, phones and social profiles per domain
- [Tech Stack Detector — Wappalyzer & BuiltWith Alternative](https://apify.com/webdatatools/tech-stack-detector) — CMS, e-commerce, analytics, pixels and payments per domain
- [Domain DNS & Email Security Checker](https://apify.com/webdatatools/dns-email-security-checker) — SPF, DKIM, DMARC, MX provider, registrar and domain age
- [Domain Security Audit (TLS, HTTP headers, redirects, robots)](https://apify.com/webdatatools/domain-security-audit) — TLS expiry, security headers, redirect chain, robots and llms.txt
- [Subdomain Finder (Certificate Transparency)](https://apify.com/webdatatools/subdomain-finder) — every subdomain seen in CT logs, with a live DNS check
- [Bulk Core Web Vitals & PageSpeed Audit](https://apify.com/webdatatools/core-web-vitals-audit) — Lighthouse scores, LCP, CLS, INP and top fixes per URL
- [On-Page SEO Audit](https://apify.com/webdatatools/seo-page-audit) — title, meta, headings, links, images and schema issues per page
- [Sitemap URL Extractor & Change Monitor](https://apify.com/webdatatools/sitemap-extractor) — every sitemap URL, or new and removed pages between runs
- [Wayback Machine Snapshot & Page Change Tracker](https://apify.com/webdatatools/wayback-page-diff) — how a page changed over time, or every archived snapshot
- [Bulk Domain WHOIS & RDAP Lookup](https://apify.com/webdatatools/domain-whois-rdap) — registrar, dates, status and nameservers per domain
- [Web Scraper — CSS Selector & Data Extractor](https://apify.com/webdatatools/css-selector-extractor) — pull any CSS selector off any page, one row per URL
- [Website Screenshot Generator](https://apify.com/webdatatools/website-screenshot) — full-page or viewport PNG/JPEG screenshots of any URL

**Content for AI, LLMs and RAG**

- [AI Web Search & Read: Google results as clean Markdown](https://apify.com/webdatatools/ai-web-search) — a query turned into clean Markdown from the top search results
- [llms.txt Generator (Website to llms.txt & llms-full.txt)](https://apify.com/webdatatools/llms-txt-generator) — llms.txt and llms-full.txt for any website or docs site
- [PDF to Markdown Converter (Text, Headings, Metadata)](https://apify.com/webdatatools/pdf-to-markdown) — PDF files to clean Markdown with headings, metadata and token counts
- [RAG Text Chunker (Markdown to Chunks, Token Counts)](https://apify.com/webdatatools/rag-text-chunker) — heading-aware RAG chunks with exact GPT token counts from any dataset
- [Article & News Extractor (clean text, author, date, markdown)](https://apify.com/webdatatools/article-extractor) — clean article text, author, date and Markdown per URL
- [Structured Data & JSON-LD Extractor (Schema.org, Open Graph)](https://apify.com/webdatatools/structured-data-extractor) — Schema.org and Open Graph data from any page
- [Google News Scraper (RSS search by keyword, topic, site)](https://apify.com/webdatatools/google-news-scraper) — news results by keyword, topic or site
- [Press Release Monitor: PR Newswire, BusinessWire, GlobeNewswire](https://apify.com/webdatatools/press-release-monitor) — PR Newswire, Business Wire and GlobeNewswire releases

**Search, video and social**

- [YouTube Shorts Scraper](https://apify.com/webdatatools/youtube-shorts-scraper) — Shorts from channels, hashtags and searches with view counts
- [Pinterest Pins Scraper](https://apify.com/webdatatools/pinterest-pins-scraper) — latest pins of public Pinterest profiles and boards
- [YouTube Transcript Scraper](https://apify.com/webdatatools/youtube-transcript-scraper) — captions and subtitles as text + timed segments, per video or channel
- [Google Search Results Scraper — SERP API](https://apify.com/webdatatools/google-search-scraper) — organic SERP results per keyword and country
- [YouTube Comments Scraper — Comments & Replies](https://apify.com/webdatatools/youtube-comments-scraper) — comments and replies with likes, no API key
- [YouTube Channel Latest Videos (RSS, no API key)](https://apify.com/webdatatools/youtube-channel-videos) — the latest 15 videos of any channel from RSS
- [YouTube Channel Scraper (videos, shorts, live)](https://apify.com/webdatatools/youtube-channel-scraper) — a channel's full video, shorts and stream list
- [YouTube Search Results Scraper (videos, channels, no API key)](https://apify.com/webdatatools/youtube-search-scraper) — videos, channels and playlists per query
- [YouTube Video Details Scraper (views, likes, description, tags)](https://apify.com/webdatatools/youtube-video-details) — views, likes, description, tags and chapters per video
- [Apple Podcasts Lookup & Episodes Scraper](https://apify.com/webdatatools/podcast-lookup) — podcast metadata and episodes from iTunes and RSS
- [Bluesky Post, Search & Profile Scraper](https://apify.com/webdatatools/bluesky-scraper) — posts, profiles, followers and threads from the AT Protocol API
- [X Tweet Scraper (Twitter Posts by URL, No Login)](https://apify.com/webdatatools/x-tweet-scraper) — X (Twitter) posts by URL with likes, replies, author, media and MP4 links
- [TikTok Profile Scraper (Followers, Likes, Bio, Video Stats)](https://apify.com/webdatatools/tiktok-profile-scraper) — TikTok profiles with exact follower and like counts, plus video stats
- [Telegram Channel Posts Scraper](https://apify.com/webdatatools/telegram-channel-scraper) — posts, views and media flags from any public channel
- [Substack Publication & Posts Scraper](https://apify.com/webdatatools/substack-scraper) — archive, authors and paywall status per publication
- [Google Play Reviews Scraper](https://apify.com/webdatatools/google-play-reviews-scraper) — reviews, ratings, replies and app versions per app
- [App Store Reviews Scraper](https://apify.com/webdatatools/app-store-reviews-scraper) — iOS reviews and ratings per app and country
- [Trustpilot Reviews Scraper (Ratings, Replies, No Login)](https://apify.com/webdatatools/trustpilot-reviews-scraper) — Trustpilot reviews with stars, full text, verification and company replies
- [Google Trends Scraper](https://apify.com/webdatatools/google-trends-scraper) — interest over time, by region, and related queries per keyword
- [Google Ads Transparency Scraper](https://apify.com/webdatatools/google-ads-transparency-scraper) — ads any advertiser runs on Google, with format and dates
- [Keyword Suggestions Scraper (Google, YouTube, Amazon, Bing)](https://apify.com/webdatatools/keyword-suggestions-scraper) — autocomplete keyword ideas from four search engines
- [Bilibili Scraper (Videos, Search, Popular)](https://apify.com/webdatatools/bilibili-scraper) — Chinese video platform: views, likes, coins, danmaku, uploader
- [Mastodon Scraper (Hashtags, Accounts, Trending)](https://apify.com/webdatatools/mastodon-scraper) — public fediverse posts by hashtag, account or trending
- [Meetup Events Scraper (Search by Keyword & City)](https://apify.com/webdatatools/meetup-events-scraper) — upcoming events with RSVPs, fees, venues and groups
- [Eventbrite Scraper (Events by Keyword & City)](https://apify.com/webdatatools/eventbrite-scraper) — events by keyword and city with venue, dates and organizer

**Leads, jobs and company data**

- [Career Site Jobs API (Greenhouse, Lever, Ashby, Workday +1)](https://apify.com/webdatatools/career-site-jobs-api) — company domains in, their open jobs out, ATS detected automatically
- [Workday Jobs Scraper](https://apify.com/webdatatools/workday-jobs-scraper) — jobs with full descriptions from any Workday career site
- [Google Maps Scraper](https://apify.com/webdatatools/google-maps-scraper) — businesses with phone, website, address, rating and coordinates per search
- [2GIS Places Scraper (Phones, Websites, Ratings, Hours)](https://apify.com/webdatatools/2gis-places-scraper) — 2GIS businesses in Russia, Kazakhstan, UAE and more with phones, website, rating and hours
- [LinkedIn Jobs Scraper](https://apify.com/webdatatools/linkedin-jobs-scraper) — job titles, companies, locations and full descriptions from LinkedIn job search
- [Company 360: full company profile from a domain](https://apify.com/webdatatools/company-360) — one row per domain: contacts, tech, security, hiring and company facts
- [Hiring Signals Scraper (Greenhouse, Lever, Ashby, Workable)](https://apify.com/webdatatools/hiring-signals) — open jobs and hiring velocity from 10 public ATS boards
- [Y Combinator Companies & Founders Scraper](https://apify.com/webdatatools/yc-companies-scraper) — YC startups by batch, industry and hiring status
- [Wikidata Entity & Company Enrichment (facts, IDs, links)](https://apify.com/webdatatools/wikidata-entity-enrichment) — HQ, founders, employees, revenue and social IDs per company
- [Email Validator & Verifier — Bulk Email Check](https://apify.com/webdatatools/email-validator) — syntax, MX, disposable, role and free-provider checks
- [OpenStreetMap POI Extractor (Overpass API: shops, amenities)](https://apify.com/webdatatools/overpass-poi-extractor) — shops and amenities by radius, bbox or area
- [Stock, Crypto & FX Quotes](https://apify.com/webdatatools/market-quotes) — one row per symbol from Yahoo, Binance and ECB rates
- [Yahoo Finance Scraper (Quotes, Fundamentals, Financials, News)](https://apify.com/webdatatools/yahoo-finance-scraper) — Yahoo Finance fundamentals, analyst targets, financial statements, ownership, news and history per ticker
- [DexScreener Scraper (DEX Pairs, Prices, Liquidity, Boosts)](https://apify.com/webdatatools/dexscreener-scraper) — DEX pairs with price, volume, liquidity, FDV and boosted tokens across chains
- [Polymarket Scraper (Prediction Markets, Odds, Volume)](https://apify.com/webdatatools/polymarket-scraper) — Polymarket markets and events with outcome prices, volume and liquidity
- [StockTwits Scraper (Messages, Sentiment, Trending)](https://apify.com/webdatatools/stocktwits-scraper) — StockTwits messages with Bullish/Bearish sentiment and trending symbols
- [Economic Calendar Scraper (Forex Events, Impact, Forecasts)](https://apify.com/webdatatools/economic-calendar-scraper) — This week's macro events with impact, forecast and previous values
- [Crypto Fear & Greed Index](https://apify.com/webdatatools/crypto-fear-greed-index) — Daily Crypto Fear & Greed Index values with full history
- [Remote Jobs Aggregator (RemoteOK, WWR, Hacker News)](https://apify.com/webdatatools/remote-jobs-aggregator) — one clean row per remote job, de-duplicated across feeds
- [Greenhouse Jobs Scraper](https://apify.com/webdatatools/greenhouse-jobs-scraper) — jobs with descriptions from any Greenhouse job board
- [Lever Jobs Scraper](https://apify.com/webdatatools/lever-jobs-scraper) — jobs with descriptions from any Lever careers page
- [Ashby Jobs Scraper](https://apify.com/webdatatools/ashby-jobs-scraper) — jobs, salaries and descriptions from any Ashby job board
- [SmartRecruiters Jobs Scraper](https://apify.com/webdatatools/smartrecruiters-jobs-scraper) — jobs with descriptions from any SmartRecruiters company
- [Seek Jobs Scraper (Australia & New Zealand)](https://apify.com/webdatatools/seek-jobs-scraper) — Seek job ads with salary, work type and location
- [Dice Jobs Scraper](https://apify.com/webdatatools/dice-jobs-scraper) — US tech jobs from Dice with salary and remote flag
- [AutoScout24 Scraper](https://apify.com/webdatatools/autoscout24-scraper) — European car listings with price, mileage and seller
- [Rightmove Scraper](https://apify.com/webdatatools/rightmove-scraper) — UK property for sale or rent with price and agent
- [Wellfound Jobs Scraper (AngelList Startup Jobs)](https://apify.com/webdatatools/wellfound-jobs-scraper) — startup jobs with salary and equity ranges, company size and stage
- [Yandex Maps Scraper (Places, Ratings, Phones)](https://apify.com/webdatatools/yandex-maps-scraper) — businesses in Russia, Türkiye and the CIS with phones, ratings, hours
- [Craigslist Scraper (Listings, Prices, Locations)](https://apify.com/webdatatools/craigslist-scraper) — listings in any area and category with price, date and coordinates
- [JobStreet Scraper (Malaysia, Singapore, PH, ID + JobsDB)](https://apify.com/webdatatools/jobstreet-scraper) — JobStreet and JobsDB jobs in 6 Asian countries with parsed salaries
- [InfoJobs Scraper (Spain Jobs, Salaries, Companies)](https://apify.com/webdatatools/infojobs-scraper) — Spanish jobs with salary range, contract type and full description
- [StepStone Scraper (Germany, Austria, Belgium Jobs)](https://apify.com/webdatatools/stepstone-scraper) — StepStone jobs in Germany, Austria and Belgium by keyword and city
- [Naukri Jobs Scraper (Salary, Skills, Company Rating)](https://apify.com/webdatatools/naukri-jobs-scraper) — Naukri.com jobs in India with salary range, experience, skills and company rating
- [Zillow Scraper (Homes for Sale, Rent & Sold, Zestimates)](https://apify.com/webdatatools/zillow-scraper) — Zillow homes for sale, rent or sold with Zestimates, beds, baths and photos
- [Zillow Home Details Scraper (Description, Photos, History)](https://apify.com/webdatatools/zillow-detail-scraper) — full Zillow property pages: description, photos, HOA, tax rate, agent
- [Redfin Scraper (Homes for Sale, Prices, Details)](https://apify.com/webdatatools/redfin-scraper) — US homes for sale or sold from any Redfin search, with price and details
- [Kleinanzeigen Scraper (Ads, Prices, Locations)](https://apify.com/webdatatools/kleinanzeigen-scraper) — German classifieds with price, VB flag, ZIP, city and seller type

**Developer, app and research data**

- [npm, PyPI & Crates.io Package Health Checker](https://apify.com/webdatatools/package-health-checker) — releases, downloads, deprecation and a health score
- [GitHub Repository Health & Activity Report](https://apify.com/webdatatools/github-repo-health) — stars, commits, contributors and risk flags per repo
- [VS Code Marketplace Extension Scraper (installs, ratings)](https://apify.com/webdatatools/vscode-marketplace-extensions) — installs, ratings and versions per extension
- [Chrome Web Store Extension Scraper (installs, ratings)](https://apify.com/webdatatools/chrome-web-store-extensions) — users, rating, version and developer per extension
- [Google Play Scraper](https://apify.com/webdatatools/google-play-scraper) — apps, ratings, installs, developer contact and reviews
- [App Store (iOS) App Metadata, Ratings & Top Charts Lookup](https://apify.com/webdatatools/app-store-lookup) — ratings, price, version and charts per app
- [CrossRef DOI & Citation Metadata Lookup](https://apify.com/webdatatools/crossref-doi-lookup) — papers, authors, journals and citation counts
- [FDA Recalls & Adverse Events Monitor (openFDA)](https://apify.com/webdatatools/openfda-recall-monitor) — food, drug and device recalls from openFDA
- [iCal / ICS Calendar Feed to Events Extractor](https://apify.com/webdatatools/ical-calendar-extractor) — any public calendar feed as event rows
- [Shopify Store Products Scraper](https://apify.com/webdatatools/shopify-products-scraper) — catalog, prices, variants and stock per store
- [Hacker News Search & Front Page Scraper](https://apify.com/webdatatools/hacker-news-scraper) — stories, comments and points by query or front page
- [GitHub Trending Repositories Scraper](https://apify.com/webdatatools/github-trending-scraper) — trending repos and developers by language and period
- [Stack Overflow & Stack Exchange Q\&A Scraper](https://apify.com/webdatatools/stackexchange-scraper) — questions, answers and scores by query, tag or site
- [Bulk Image Downloader](https://apify.com/webdatatools/bulk-image-downloader) — download image URLs to storage with size, dimensions and a ZIP
- [Google Flights Scraper (Prices, Airlines, Stops)](https://apify.com/webdatatools/google-flights-scraper) — flight prices, airlines, times, stops and CO2 by route and date
- [Google Hotels Scraper (Prices, Ratings, Reviews)](https://apify.com/webdatatools/google-hotels-scraper) — hotel prices per night, stars, rating and reviews by city and dates
- [Booking.com Scraper (Hotel Prices, Ratings, Availability)](https://apify.com/webdatatools/booking-scraper) — Booking.com hotels for any city and dates with prices, scores and deals
- [Airbnb Scraper (Listings, Prices, Ratings, Coordinates)](https://apify.com/webdatatools/airbnb-scraper) — Airbnb stays for any city and dates with prices, ratings and coordinates
- [AliExpress Scraper (Search Products & Prices)](https://apify.com/webdatatools/aliexpress-scraper) — AliExpress search results with USD price, discount and rank
- [Lazada Scraper (Products, Prices, Sold, Ratings)](https://apify.com/webdatatools/lazada-scraper) — Lazada products in 6 countries with price, rating, units sold and seller
- [Trendyol Scraper (Turkey Products, Prices, Ratings)](https://apify.com/webdatatools/trendyol-scraper) — Trendyol products with price, basket discount, rating and promotions

### Support and feedback

Found a bug or want a feature? Open an issue on the **Issues** tab.

URL deduplication normalizes the host and removes fragments while preserving case-sensitive paths and query values. `/Docs` and `/docs` stay separate in start URLs, discovered links, and sitemaps. Every supplied domain participates in the crawl scope; an unrelated third domain is excluded when `sameDomainOnly` is enabled.

# Actor input Schema

## `url` (type: `string`):

Optional, for AI agents and API calls. Send only {"url": "https://…"} to get that one page as Markdown in seconds: no sitemap, no link following, 1 result (MCP call-actor waits at most 45 s). When set, Start URLs and Max pages are ignored; leave it empty to crawl a site from Start URLs.

## `startUrls` (type: `array`):

Enter the page(s) to start crawling from, e.g. https://docs.apify.com/platform. The crawler follows links from here in breadth-first order. Each page is saved to the dataset as soon as it is converted.

## `maxPages` (type: `integer`):

Enter the maximum number of pages to crawl and convert, e.g. 50. This is the billed unit ($1 per 1,000 pages) — the crawl stops exactly at this count. A typical site crawls at 2–10 pages per second; if the run timeout comes first, the crawl stops early and the run still ends SUCCEEDED with every page done so far.

## `maxDepth` (type: `integer`):

Enter how many links deep to follow from a start URL, e.g. 3. Set 0 to crawl only the start URL(s).

## `sameDomainOnly` (type: `boolean`):

Turn this on to follow links only on any supplied start domain (www. counts as the same). With multiple start domains, each is allowed; other domains are excluded. Turn it off to allow other domains.

## `includePathPrefixes` (type: `array`):

Optional. Only crawl URLs whose path starts with one of these prefixes, e.g. /docs. Leave empty to crawl every path allowed by the other settings.

## `excludePathPatterns` (type: `array`):

Optional. Regular expressions tested against the full URL; a match is skipped, e.g. .(png|jpe?g)$ to skip images. Defaults cover binary files and login/signup pages.

## `useSitemap` (type: `boolean`):

Turn this on to also read /sitemap.xml on each start URL's domain and add its URLs (filtered by the settings above) to the crawl queue, up to Max pages.

## `outputFormat` (type: `string`):

Choose what content to put in each row: Markdown only, plain text only, or both. Markdown preserves headings, lists, code blocks and tables.

## `removeSelectors` (type: `array`):

Optional. Extra CSS selectors to strip before extracting content, e.g. .cookie-banner or #newsletter-signup, on top of the built-in nav/header/footer/aside removal.

## `maxConcurrency` (type: `integer`):

Enter the maximum number of pages fetched in parallel, e.g. 10. Lower it if the target site rate-limits you.

## `maxPagesPerDomain` (type: `integer`):

Optional. Cap how many pages are crawled from any one domain (www. ignored), e.g. 20 when you crawl many sites or follow links off-site. Leave empty for no per-domain cap; Max pages still caps the run.

## `maxConsecutiveFailures` (type: `integer`):

Stop the crawl when this many pages fail in a row (blocked site, dead host) instead of working through the whole queue. Failed pages are never charged. Default 20; set 0 to never stop for failures.

## `respectRobots` (type: `boolean`):

Turn this on to skip URLs disallowed by the site's robots.txt file (recommended and on by default).

## `proxyConfiguration` (type: `object`):

Optional. Select a proxy configuration, e.g. Apify Proxy with the datacenter group, if the target site blocks requests from shared IPs.

## `chunkForRag` (type: `boolean`):

Turn this on to add a `chunks` array to every page: heading-aware pieces of the Markdown with the heading path and an exact GPT (o200k) token count, ready to embed. Same price — chunks are part of the page result.

## `chunkSize` (type: `integer`):

Enter the maximum tokens per chunk when Add RAG chunks is on, e.g. 500.

## `changeDetection` (type: `boolean`):

Turn this on to compare every page with the previous run that used the same start URLs and add `changeStatus`: new, changed or unchanged. Ideal for a scheduled run that keeps a vector DB fresh.

## `skipUnchangedContent` (type: `boolean`):

With change detection on, unchanged pages keep their row (url, title, changeStatus) but get empty markdown/text/chunks so you only re-embed what changed. Each page is still one result.

## `inputDatasetId` (type: `string`):

Take the URLs from another Actor run's dataset, e.g. "aBcD1234". Combined with the list above.

## `inputField` (type: `string`):

Column holding the value, e.g. "url" (the default) or "website". Array columns are flattened.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    },
    {
      "url": "https://crawlee.dev/docs"
    }
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "includePathPrefixes": [],
  "excludePathPatterns": [
    "\\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$",
    "/login",
    "/signup",
    "\\?replytocom="
  ],
  "useSitemap": true,
  "outputFormat": "markdown",
  "removeSelectors": [],
  "maxConcurrency": 10,
  "maxConsecutiveFailures": 20,
  "respectRobots": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "chunkForRag": false,
  "chunkSize": 500,
  "changeDetection": false,
  "skipUnchangedContent": true
}
```

# Actor output Schema

## `pages` (type: `string`):

All crawled pages — download as JSON, CSV, Excel or HTML.

## `runSummary` (type: `string`):

Confirmed rows, source and discovery failures, partial output, budget limits and unconfirmed delivery.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform"
        },
        {
            "url": "https://crawlee.dev/docs"
        }
    ],
    "maxPages": 50,
    "maxDepth": 3,
    "includePathPrefixes": [],
    "excludePathPatterns": [
        "\\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$",
        "/login",
        "/signup",
        "\\?replytocom="
    ],
    "outputFormat": "markdown",
    "removeSelectors": [],
    "maxConcurrency": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("webdatatools/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://docs.apify.com/platform" },
        { "url": "https://crawlee.dev/docs" },
    ],
    "maxPages": 50,
    "maxDepth": 3,
    "includePathPrefixes": [],
    "excludePathPatterns": [
        "\\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$",
        "/login",
        "/signup",
        "\\?replytocom=",
    ],
    "outputFormat": "markdown",
    "removeSelectors": [],
    "maxConcurrency": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("webdatatools/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    },
    {
      "url": "https://crawlee.dev/docs"
    }
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "includePathPrefixes": [],
  "excludePathPatterns": [
    "\\\\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$",
    "/login",
    "/signup",
    "\\\\?replytocom="
  ],
  "outputFormat": "markdown",
  "removeSelectors": [],
  "maxConcurrency": 10
}' |
apify call webdatatools/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,webdatatools/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XzLicaoE1OdrHNA4E/builds/UDxevY7fvbGzuNB2i/openapi.json