# Website to Markdown Crawler | Pre-Chunked for RAG (`themineworks/rag-crawler`) Actor

Crawl any website into clean, pre-chunked Markdown with per-chunk token counts for RAG pipelines, vector DBs (Pinecone, Qdrant) and LLM context. MCP native for Claude & ChatGPT. SPA support via Playwright. Pay only for pages that crawl. A Firecrawl alternative.

- **URL**: https://apify.com/themineworks/rag-crawler.md
- **Developed by:** [The Mine Works](https://apify.com/themineworks) (community)
- **Categories:** AI, Developer tools, MCP servers
- **Features:** Limited permissions
- **Stats:** 7 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler | Pre-Chunked for RAG

[![29 docs pages to 303 RAG chunks in 26 s](https://api.apify.com/v2/key-value-stores/cUXz95yxflDho41nn/records/rag-crawler-hero.png)](https://console.apify.com/actors/a3WCCqmKCdCjIZZAb/input?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

From **The Mine Works**, makers of [Threads Scraper](https://apify.com/themineworks/threads-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral) and [B2B Leads Finder](https://apify.com/themineworks/b2b-leads-finder?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral), with over 290,000 runs across 310+ public actors.

Point it at a website and get every page back as clean Markdown, already cut into chunks along its headings, each chunk with its heading path and a token estimate. That is the shape a RAG pipeline, a vector database (Pinecone, Qdrant, Weaviate) or an LLM context window wants, without writing HTML cleanup or chunking code. Plain HTTP by default, a real browser when a site needs JavaScript. No login, no API key.

### Why choose this actor?

- **A docs site turned into chunks in under half a minute.** In a recorded run on 2 October 2026, Apify's documentation (from `https://docs.apify.com/platform`, 30 pages requested) came back as **29 pages and 303 chunks in 26 seconds**, about 30,000 tokens in all, with no chunk over the 500-token limit we set.
- **Chunks that keep their place in the document.** Every chunk carries `heading_path`, the headings above it from the top level down (two levels deep on Apify's schedules page, for example), and a section split into several chunks repeats its heading at the top of each, so retrieval keeps its context.
- **Pay only for pages that worked.** A page that times out, fails to render, is blocked or has no readable text comes back as a free `failed` row with the reason. The Gold price is $1.00 per 1,000 pages.

[![Run it on Apify](https://api.apify.com/v2/key-value-stores/cUXz95yxflDho41nn/records/button-run.png)](https://console.apify.com/actors/a3WCCqmKCdCjIZZAb/input?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Part of The Mine Works Developer and AI tools family:** [GitHub Skill Finder](https://apify.com/themineworks/github-skill-discovery?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral), [GitHub Repo Scraper](https://apify.com/themineworks/github-repo-intelligence?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral), [GitHub Trending Scraper](https://apify.com/themineworks/github-trending-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral).

### Try it in one minute

Paste this into the JSON tab of the input page and press Start. It crawls 10 pages of a documentation site in well under a minute.

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "maxPages": 10,
  "outputFormat": "both"
}
```

The only input you must give is `startUrls`: one or more pages to start from, each as `{ "url": "https://..." }` (in the Console you can paste URLs one per line). The crawler follows links from those pages within the same domain until it reaches `maxPages`. Use `excludeUrlPatterns` to keep it out of sections you do not want, and `outputFormat` to choose chunks, full-page Markdown or both.

Apify's free plan includes $5 of credit every month, which covers about 2,400 pages at this actor's Free plan price ($0.002 a page plus the $0.005 start fee, in runs of 100 pages).

#### Copy to your AI assistant

Paste this block into ChatGPT, Claude, Cursor or any assistant that can write code, and it can run the actor for you.

```
themineworks/rag-crawler on Apify. Crawls a website from start URLs within the same domain and returns one row per page with url, canonical, title, description, language, word_count, token_count, crawled_at, status, plus chunks (array of {heading_path, text, token_count, chunk_index}) and/or markdown. Call ApifyClient("TOKEN").actor("themineworks/rag-crawler").call(run_input={...}), then client.dataset(run["defaultDatasetId"]).list_items().items. Required: startUrls ([{"url": "https://..."}]). Optional: maxPages (1 to 500, default 10), renderJs (default false; true uses a headless browser), waitStrategy ("networkidle" default, "domcontentloaded" or "load", browser mode only), outputFormat ("chunks" default, "full" or "both"), maxTokensPerChunk (50 to 4000, default 800), stripBoilerplate (default true), excludeUrlPatterns (regex strings), customCss (CSS selector for the main content). Rows with status "failed" are not billed; rows with _type "summary" or "info" are run reports. Full spec: GET https://api.apify.com/v2/acts/themineworks~rag-crawler/builds/default (Bearer TOKEN), which returns inputSchema and readme. Token: https://console.apify.com/account/integrations?fpr=ymnoit&utm_source=apify-readme&utm_medium=referral
```

### Key features

- **Up to 500 pages per run**, following links within the start URL's domain. In our test, 30 page requests produced 29 pages in 26 seconds on plain HTTP.
- **Heading-aware chunking.** Markdown is split at every heading level (`#` to `######`); a section longer than `maxTokensPerChunk` is packed paragraph by paragraph into several chunks, each starting with the section's heading line.
- **Three output shapes.** `chunks` (default) for RAG, `full` for one Markdown document per page, or `both`.
- **Boilerplate removal.** Mozilla Readability picks the main article, and navigation, headers, footers, sidebars, forms, scripts and common ad or cookie elements are dropped. A `customCss` selector, when it matches, overrides all of that and keeps exactly the element you name.
- **JavaScript sites.** `renderJs: true` switches to a headless browser (Playwright) with a choice of when to read the page (`waitStrategy`).
- **Token estimates for budgeting.** Every page and every chunk carries `token_count`, an estimate of words times 1.35, so you can size embedding costs and context windows before indexing.

### How to use it

#### Basic: one site, default settings

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "maxPages": 50
}
```

You get up to 50 pages as heading-split chunks of at most about 800 tokens each.

#### Tight chunks for embeddings, plus the full page

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "maxPages": 30,
  "outputFormat": "both",
  "maxTokensPerChunk": 500
}
```

This is the exact input of our 2 October test: 29 pages, 303 chunks, no chunk over 500 estimated tokens, in 26 seconds. Index `chunks`; keep `markdown` for display or re-chunking later.

#### Keep the crawl inside one section

```json
{
  "startUrls": [{ "url": "https://docs.example.com/guides/" }],
  "maxPages": 200,
  "excludeUrlPatterns": ["/blog/", "/changelog/", "/api-reference/", "\\.(pdf|zip|png|jpg|jpeg|gif|svg)$"]
}
```

The crawler follows any link on the same domain, including other subdomains of it (in our test it reached `console.apify.com` from `docs.apify.com`). List the sections you do not want as regular expressions; matching links are never queued and never charged.

#### A JavaScript-only site

```json
{
  "startUrls": [{ "url": "https://app.example.com/help" }],
  "maxPages": 25,
  "renderJs": true,
  "waitStrategy": "networkidle"
}
```

Use this only when the plain mode returns `no_content` failures or empty pages, since a browser is slower. `domcontentloaded` is quicker than `networkidle` when the content is in the first HTML.

#### Nightly refresh of a help center for a support bot

```json
{
  "startUrls": [{ "url": "https://help.example.com/" }],
  "maxPages": 300,
  "outputFormat": "chunks",
  "customCss": "article"
}
```

Save this as a task and schedule it nightly (for example `0 2 * * *`), then upsert chunks into your vector store keyed on `url` plus `chunk_index`. `customCss` keeps only the matched element when it exists on a page, which is the cleanest option for a site with one consistent article template.

### Input parameters

| Parameter | Type | Default | What it does |
| --- | --- | --- | --- |
| `startUrls` | array | required (form prefill: `https://example.com`) | Pages to start from, as `{ "url": "..." }` objects. |
| `maxPages` | integer (1 to 500) | `10` | Most pages to request in the run, including failed ones. |
| `renderJs` | boolean | `false` | `false` fetches HTML directly (fast); `true` renders each page in a headless browser. |
| `waitStrategy` | string | `networkidle` | Browser mode only: `networkidle`, `domcontentloaded` or `load`. If the chosen event does not come within 30 seconds, the actor falls back to `domcontentloaded`. |
| `outputFormat` | string | `chunks` | `chunks` adds the `chunks` array, `full` adds `markdown`, `both` adds both. |
| `maxTokensPerChunk` | integer (50 to 4,000) | `800` | Target size of a chunk, in estimated tokens. A single paragraph longer than this is never split, so the odd chunk can be larger. |
| `stripBoilerplate` | boolean | `true` | Use Mozilla Readability to keep the main content. Off, the whole page body is converted (navigation and footers are still dropped). |
| `excludeUrlPatterns` | array of strings | `[]` (form prefill: file extensions, `/tag/`, `/author/`) | Regular expressions, case-insensitive. Matching links are skipped. |
| `customCss` | string | empty (form prefill: `article.main-content`) | A CSS selector for the main content. When it matches, only that element is converted; when it does not, the actor falls back to Readability. |

"Form prefill" values fill the Console form for you; an API call that leaves a field out gets the default shown. The Console prefill for `customCss` is only an example; it is ignored on pages where nothing matches.

There is no proxy setting: pages are fetched directly from Apify's servers, which suits public documentation and content sites. The default timeout is 300 seconds; for runs of a few hundred pages, or any run with `renderJs`, give the run more time in Run options.

### What data do you get?

One row per page.

**The page**: `url`, `canonical` (from the page's canonical link, or the URL itself when there is none), `title`, `description` (the meta description), `language` (the `lang` attribute of the page's `<html>` tag), `crawled_at`, `status` (`success`).

**Size**: `word_count` and `token_count` for the whole cleaned page. `token_count` is an estimate (words times 1.35, rounded up), not the count of any one model's tokenizer; expect it to be within about 10% for English prose and further off for code.

**Content**: `chunks` (with `outputFormat` `chunks` or `both`), an array of `{ heading_path, text, token_count, chunk_index }`, and `markdown` (with `full` or `both`), the whole cleaned page. `heading_path` is the list of headings above the chunk, empty for text before the first heading; `chunk_index` counts from 0 within the page. Headings are copied as the page writes them, so documentation sites that put anchor links in their headings keep those (for example `Apify Console[](#apify-console)`).

`description` and `language` are read from the raw HTML and need quoted attributes: on Wikipedia `language` came back as `en`, but on the minified Apify docs both were empty strings on all 29 pages.

**Failed pages** come back as `{ url, status: "failed", reason, message, charged: false }`, where `reason` is `no_content`, `timeout`, `render_error` or `blocked`. They are never billed.

Each run ends with a `_type: "summary"` row: `pages_requested`, `pages_crawled`, `pages_failed`, `total_tokens`, `charged_for` and the run's own cost-guard figures; a `_type: "info"` row follows when pages were delivered. Neither is billed.

#### Stable fields for automations

These fields were present in every successful page row we sampled (32 pages across two runs on 2 October 2026):

| Field | What it holds |
| --- | --- |
| `url` | The page address that was crawled |
| `canonical` | Canonical URL, or `url` when the page declares none |
| `title` | Page title |
| `description` | Meta description, or an empty string |
| `language` | `<html lang>` value, or an empty string |
| `word_count` | Words in the cleaned Markdown |
| `token_count` | Estimated tokens in the cleaned Markdown |
| `crawled_at` | ISO timestamp of the crawl |
| `status` | `success` on page rows (`failed` rows are described above) |
| `chunks` | Present when `outputFormat` is `chunks` or `both` |
| `markdown` | Present when `outputFormat` is `full` or `both` |

Inside `chunks`, every item has `heading_path`, `text`, `token_count` and `chunk_index`. We will not rename these fields. New fields may be added over time; existing ones keep their names.

#### Output examples

Real rows from our 2 October 2026 runs, with text trimmed.

A documentation page with nested headings (`outputFormat: "both"`, `maxTokensPerChunk: 500`; 8 chunks, the first three shown):

```json
{
  "url": "https://docs.apify.com/actors/running/schedules",
  "canonical": "https://docs.apify.com/actors/running/schedules",
  "title": "Actor and task schedules | Platform | Apify Documentation",
  "description": "",
  "language": "",
  "word_count": 1278,
  "token_count": 1726,
  "crawled_at": "2026-10-02T08:25:21.241Z",
  "status": "success",
  "chunks": [
    {
      "heading_path": [],
      "text": "Schedules allow you to run your Actors and tasks at specific times. You schedule the run frequency using [cron expressions](#cron-expressions)…",
      "token_count": 185,
      "chunk_index": 0
    },
    {
      "heading_path": ["Set up a new schedule[](#set-up-a-new-schedule)"],
      "text": "## Set up a new schedule[](#set-up-a-new-schedule)\n\nBefore setting up a new schedule, you should have the [Actor](https://docs.apify.com/actors) or [task](https…",
      "token_count": 126,
      "chunk_index": 1
    },
    {
      "heading_path": ["Set up a new schedule[](#set-up-a-new-schedule)", "Apify Console[](#apify-console)"],
      "text": "### Apify Console[](#apify-console)\n\nIn [Apify Console](https://console.apify.com/schedules?fpr=ymnoit&utm_source=apify-readme&utm_medium=referral), click on the **Schedules** in the navigation menu…",
      "token_count": 424,
      "chunk_index": 2
    }
  ],
  "markdown": "Schedules allow you to run your Actors and tasks at specific times. You schedule the run frequency using [cron expressions](#cron-expressions).\n\nTimezone & Daylight Savings Time…"
}
```

A Wikipedia article (`outputFormat: "chunks"`, default 800-token limit; 14 chunks, two shown):

```json
{
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "canonical": "https://en.wikipedia.org/wiki/Web_scraping",
  "title": "Web scraping - Wikipedia",
  "description": "",
  "language": "en",
  "word_count": 4523,
  "token_count": 6107,
  "crawled_at": "2026-10-02T08:26:10.977Z",
  "status": "success",
  "chunks": [
    {
      "heading_path": ["Human copy-and-paste"],
      "text": "### Human copy-and-paste\n\n\\[[edit](https://en.wikipedia.org/w/index.php?title=Web_scraping&action=edit&section=3 …",
      "token_count": 134,
      "chunk_index": 2
    },
    {
      "heading_path": ["Methods to prevent web scraping"],
      "text": "## Methods to prevent web scraping\n\n- [Academic journal publishing reform](https://en.wikipedia.org/wiki/Academic_journa…",
      "token_count": 168,
      "chunk_index": 11
    }
  ]
}
```

The `title` here is Wikipedia's own page title, which contains its own hyphen. Note also that Wikipedia's "\[edit]" links survive the cleanup; strip them in your pipeline if they matter to you.

The smallest possible run, from our daily platform check on 1 October (example.com, one page, 22 seconds), with its summary row:

```json
{
  "url": "https://example.com",
  "canonical": "https://example.com",
  "title": "Example Domain",
  "description": "",
  "language": "",
  "word_count": 27,
  "token_count": 37,
  "crawled_at": "2026-10-01T09:00:49.448Z",
  "status": "success",
  "chunks": [
    {
      "heading_path": [],
      "text": "This domain is for use in documentation examples without needing permission…",
      "token_count": 37,
      "chunk_index": 0
    }
  ]
}
```

```json
{
  "_type": "summary",
  "pages_requested": 1,
  "pages_crawled": 1,
  "pages_failed": 0,
  "total_tokens": 37,
  "charged_for": 1
}
```

### Pricing

Pay per event: you pay for each page crawled and delivered to your dataset, plus a small start fee per run.

| Event | Free | Bronze | Silver | Gold and above |
| --- | --- | --- | --- | --- |
| `page-crawled`, per page | $0.002 | $0.0016 | $0.00125 | $0.001 |
| `page-crawled`, per 1,000 pages | $2.00 | $1.60 | $1.25 | $1.00 |
| `apify-actor-start`, per run | $0.005 per GB of run memory, minimum one event | same | same | same |

**The start fee, exactly.** Apify's `apify-actor-start` event is charged once when a run starts, at $0.005 for each GB of memory the run uses, with a minimum of one event. This actor runs on 1 GB by default, so a default run pays $0.005. If you raise memory for browser mode (for example to 4 GB), the start fee rises to $0.02.

**Worked examples on the Gold tier:** 30 pages cost $0.03 plus $0.005. 500 pages, the most in one run, cost $0.50 plus $0.005. On the Free tier, 500 pages cost $1.00 plus $0.005. The price is per page, not per chunk: the 303 chunks of our test cost the same as 29 pages.

**Never charged:** pages that time out, fail to render, are blocked or have no readable text (the `failed` rows), links skipped by `excludeUrlPatterns`, retries, and the `summary` and `info` rows. A run where every page fails pays only the start fee, and the crawl stops after 5 failed pages in a row.

There is no scheduled price change for this actor; these prices have applied since 14 September 2026. The Pricing tab on this page always shows the rate for your own plan; if it and this table ever differ, the Pricing tab is right.

### Run it on a schedule

Turn on `monitorMode` and each run delivers only the pages that are new or have changed since an earlier run with the same input, so a daily run costs you only for what moved.

1. Enter your input, switch on **Monitor mode** and click **Save as a task**.
2. In Apify Console open **Schedules**, click **Add schedule** and pick **Daily** (or any time and timezone you like).
3. Under **Actors or tasks to run**, add the task you saved and save the schedule.

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ],
  "maxPages": 20,
  "outputFormat": "chunks",
  "monitorMode": true
}
```

The first run delivers every page it finds. After that, a page is delivered again only when one of its fields changed, and unchanged pages are skipped and never charged. The summary row at the end shows `new_this_run`, `changed_this_run` and `skipped_duplicates`. Changing the input starts a fresh history; changing only the result limit does not. A changed page comes back whole, with all of its chunks, so you can replace that page's chunks in your vector store. Unchanged pages are still crawled, so changes deeper in the site are found, but they are not delivered and cost you nothing. A site where little changed can stop before `maxPages` to keep request costs in line; the summary row then shows `profit_guard_tripped`.

### FAQ

**What does it crawl?**
Any public website. It starts at your `startUrls`, follows links on the same domain (including its other subdomains) and stops at `maxPages`. It does not log in, and it reads only what a visitor can see.

**How accurate are the token counts?**
They are estimates: the word count times 1.35, rounded up. That is close for English prose and rougher for code or other languages. Use them for budgeting and chunk sizing; if you need exact counts for a specific model, run that model's tokenizer on `text`.

**How does the chunking work?**
The cleaned Markdown is split at every heading. A section that fits within `maxTokensPerChunk` becomes one chunk; a longer one is packed paragraph by paragraph into several chunks that each start with the section's heading. A paragraph is never cut in half, so one very long paragraph or list can produce a chunk over the limit (one of 14 chunks on the Wikipedia page in our test, at 1,201 estimated tokens against 800).

**Does it work on JavaScript-heavy sites?**
Yes, with `renderJs: true`, which loads each page in a headless browser before converting it. Browser mode is slower and uses more compute, so try the default plain mode first; our tests on 2 October used plain mode.

**Which proxy does it use?**
None. Pages are requested directly from Apify's servers, which works for public documentation and content sites. Sites that block datacenter traffic will show up as `blocked` or `timeout` rows, which are free.

**How fresh is the data?**
Every run fetches the pages live. Nothing is cached between runs.

**How do I keep the crawl focused?**
Start from the section you want, set `maxPages`, and list unwanted paths in `excludeUrlPatterns` as regular expressions (for example `/blog/`, `/tag/` or `\.(pdf|zip)$`; inside JSON the backslash is written twice). For a site with one consistent article template, `customCss` gives the cleanest text.

**Can I run it on a schedule?**
Yes. Save your input as a task, then in Apify Console open Schedules, create a schedule (for example nightly, `0 2 * * *`) and pick the task. Each run crawls the pages again; key your vector store on `url` and `chunk_index` and replace a page's chunks when it is re-crawled.

**How do I export the data?**
From the run's Storage tab as JSON, CSV, Excel, XML or HTML, or through the Apify API. JSON keeps `chunks` intact; in CSV they are flattened into numbered columns. Skip rows with `_type` or `status: "failed"` if you want pages only.

**Can I use it from Claude, ChatGPT or another AI assistant?**

- Connector URL: `https://mcp.apify.com/?tools=themineworks/rag-crawler`.
- Claude: Settings > Connectors > Add custom connector, paste the URL, sign in with Apify.
- ChatGPT: developer mode, add an MCP connector with the URL, sign in with Apify.
- Cursor or VS Code: add it as an HTTP MCP server with that URL.
- Claude Code: `claude mcp add -t http rag-crawler "https://mcp.apify.com/?tools=themineworks/rag-crawler"`.

**Is it legal to crawl websites this way?**
The actor reads only public pages and does not get past logins or paywalls. Website content is usually copyrighted, and many sites set terms or robots rules for automated access, so crawl sites you own, sites whose terms allow it, or content you are licensed to use, and respect data protection laws such as GDPR and CCPA for any personal data on the pages. You are responsible for how you use the data. This is general information, not legal advice.

### Integrations

- **Vector databases:** load `chunks` into Pinecone, Qdrant, Weaviate or pgvector from a webhook or a short script on the dataset.
- **Make, Zapier and n8n:** use the Apify app or node to run a crawl and pass the chunks to an embedding step.
- **Google Sheets:** export a run to a sheet to review titles, word counts and failures.
- **Webhooks:** have Apify call your URL when a run succeeds, then fetch the dataset.
- **API and client libraries:** start runs and read datasets from Python, JavaScript or any HTTP client. See the "Copy to your AI assistant" block above for the exact call.
- **MCP clients:** Claude, ChatGPT, Cursor, VS Code and other MCP clients can call the actor through `https://mcp.apify.com`.

### More from The Mine Works

**Developer and AI tools**

- [GitHub Skill Finder](https://apify.com/themineworks/github-skill-discovery?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [GitHub Repo Scraper](https://apify.com/themineworks/github-repo-intelligence?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [GitHub Trending Scraper](https://apify.com/themineworks/github-trending-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Social media and video**

- [Threads Scraper](https://apify.com/themineworks/threads-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Reddit Scraper](https://apify.com/themineworks/reddit-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Threads Search Scraper](https://apify.com/themineworks/threads-search-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Instagram Profile Scraper](https://apify.com/themineworks/instagram-profile-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Leads and business directories**

- [B2B Leads Finder](https://apify.com/themineworks/b2b-leads-finder?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Google Maps Email Scraper](https://apify.com/themineworks/maps-leads?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Skip Trace Lookup](https://apify.com/themineworks/skip-trace-lookup?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [JustDial Scraper](https://apify.com/themineworks/justdial-business?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Marketing, SEO and reviews**

- [Facebook Ad Library Scraper](https://apify.com/themineworks/meta-ad-library-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Google Ads Transparency Scraper](https://apify.com/themineworks/google-ads-transparency?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Similarweb Scraper](https://apify.com/themineworks/similarweb-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Google News Scraper](https://apify.com/themineworks/google-news?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**LinkedIn**

- [LinkedIn Company Scraper](https://apify.com/themineworks/linkedin-company-details?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [LinkedIn Post Scraper](https://apify.com/themineworks/linkedin-post-search?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [LinkedIn Employees Scraper](https://apify.com/themineworks/linkedin-employees?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [LinkedIn Profile Scraper](https://apify.com/themineworks/linkedin-profile-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Real estate**

- [Zillow Rentals Scraper](https://apify.com/themineworks/zillow-rental-listings?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Zillow Sold Comps Scraper](https://apify.com/themineworks/zillow-recently-sold?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Housing.com Scraper](https://apify.com/themineworks/housing-com-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [India Real Estate MCP](https://apify.com/themineworks/india-real-estate-mcp?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Science, health and government data**

- [CourtListener Scraper](https://apify.com/themineworks/courtlistener-court-records?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [data.gov.in Scraper](https://apify.com/themineworks/india-data-gov-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Socrata Open Data Scraper](https://apify.com/themineworks/socrata-open-data?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Academic Research MCP](https://apify.com/themineworks/academic-research-mcp?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Jobs and hiring**

- [Foundit Monster India Jobs](https://apify.com/themineworks/foundit-jobs-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Hirist Jobs Scraper](https://apify.com/themineworks/hirist-jobs-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [India Jobs MCP](https://apify.com/themineworks/india-jobs-mcp?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Naukri Jobs Scraper](https://apify.com/themineworks/naukri-jobs?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**E-commerce and marketplaces**

- [Ozon.ru Scraper](https://apify.com/themineworks/ozon-product-search?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [⭐ Amazon Reviews Scraper](https://apify.com/themineworks/amazon-reviews?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Amazon Product Scraper](https://apify.com/themineworks/amazon-products?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Carsales.com.au Scraper](https://apify.com/themineworks/carsales-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Company and business data**

- [Company Domain Finder](https://apify.com/themineworks/company-domain-finder?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [GST Taxpayer Lookup](https://apify.com/themineworks/gst-taxpayer-lookup?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [World Bank Trade Scraper](https://apify.com/themineworks/global-trade-data?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Company KYB Resolver](https://apify.com/themineworks/company-identity-resolver?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**Food and local services**

- [NoBroker Scraper](https://apify.com/themineworks/nobroker-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Swiggy Restaurant Scraper](https://apify.com/themineworks/swiggy-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Zomato Scraper](https://apify.com/themineworks/zomato-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

**More tools**

- [Tennis Match & Player Data Scraper](https://apify.com/themineworks/tennis-match-data?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Google Hotels Prices Scraper](https://apify.com/themineworks/google-hotels-prices-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [LandWatch Scraper](https://apify.com/themineworks/landwatch-land-for-sale-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)
- [Taobao Products Scraper 淘宝 天猫](https://apify.com/themineworks/taobao-products-scraper?fpr=ymnoit\&utm_source=apify-readme\&utm_medium=referral)

### Support

Found a bug or need a field we do not return yet? Open an issue on the Issues tab of this page and we will reply there. To ask for a new source, email dmineworks@gmail.com. A guide for this actor lives at [themineworks.com](https://themineworks.com/actors/rag-crawler/), with a comparison in [Firecrawl vs RAG Crawler](https://themineworks.com/blog/firecrawl-vs-rag-crawler-comparison/).

*Website to Markdown Crawler turns a website into clean, heading-aware Markdown chunks for RAG, billed only for the pages it actually crawls.*

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs to start crawling from. Each entry is a Crawlee RequestOptions object (at minimum {url: 'https://...'}), e.g. {"url": "https://docs.example.com"}.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl. You will only be charged for pages that successfully crawl.

## `renderJs` (type: `boolean`):

Use Playwright for JS-rendered SPAs. Disable for static HTML — faster and cheaper.

## `waitStrategy` (type: `string`):

Playwright wait event before extracting content. networkidle is most reliable for SPAs; domcontentloaded is faster for static pages.

## `outputFormat` (type: `string`):

chunks: heading-split chunks with token counts (ideal for RAG). full: single markdown blob per page. both: include both.

## `maxTokensPerChunk` (type: `integer`):

Target maximum tokens per chunk when outputFormat is 'chunks' or 'both'. Oversized heading sections are split by paragraph.

## `stripBoilerplate` (type: `boolean`):

Use Mozilla Readability to remove navigation, headers, footers, ads and sidebars before converting to markdown.

## `excludeUrlPatterns` (type: `array`):

List of regex patterns. Any discovered URL matching any pattern is skipped and never enqueued, e.g. "/tag/" or "/author/" to skip listing and author pages.

## `customCss` (type: `string`):

Optional CSS selector to scope content extraction (e.g. 'article.main-content', '#docs-body'). Overrides Readability when set.

## `monitorMode` (type: `boolean`):

Run on a schedule and deliver ONLY pages that are new or changed since a previous run. You are charged per delivered page, not for unchanged ones. Ideal for daily monitoring. Requires the same input across runs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxPages": 10,
  "renderJs": false,
  "waitStrategy": "networkidle",
  "outputFormat": "chunks",
  "maxTokensPerChunk": 800,
  "stripBoilerplate": true,
  "excludeUrlPatterns": [
    "\\.(pdf|zip|png|jpg|jpeg|gif|svg|mp4|mp3|woff2?)$",
    "/tag/",
    "/author/"
  ],
  "customCss": "article.main-content",
  "monitorMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ],
    "excludeUrlPatterns": [
        "\\.(pdf|zip|png|jpg|jpeg|gif|svg|mp4|mp3|woff2?)$",
        "/tag/",
        "/author/"
    ],
    "customCss": "article.main-content"
};

// Run the Actor and wait for it to finish
const run = await client.actor("themineworks/rag-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://example.com" }],
    "excludeUrlPatterns": [
        "\\.(pdf|zip|png|jpg|jpeg|gif|svg|mp4|mp3|woff2?)$",
        "/tag/",
        "/author/",
    ],
    "customCss": "article.main-content",
}

# Run the Actor and wait for it to finish
run = client.actor("themineworks/rag-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "excludeUrlPatterns": [
    "\\\\.(pdf|zip|png|jpg|jpeg|gif|svg|mp4|mp3|woff2?)$",
    "/tag/",
    "/author/"
  ],
  "customCss": "article.main-content"
}' |
apify call themineworks/rag-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,themineworks/rag-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/a3WCCqmKCdCjIZZAb/builds/CPvjJxLxXWtogknfX/openapi.json