# Docs Markdown Rag Ready Crawler (`devwithbobby/docs-markdown-rag-ready-crawler`) Actor

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

- **URL**: https://apify.com/devwithbobby/docs-markdown-rag-ready-crawler.md
- **Developed by:** [Dev with Bobby](https://apify.com/devwithbobby) (community)
- **Categories:** AI, Agents
- **Features:** Limited permissions
- **Stats:** 10 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs Markdown RAG-Ready Crawler

An Apify Actor that crawls documentation websites and converts them into clean markdown with RAG-ready chunks for embeddings. Includes internal link graphs and content hashes for change detection.

### Features

- **Markdown Conversion** - Converts HTML content to clean, well-formatted markdown
- **RAG-Ready Chunks** - Automatically splits content into chunks optimized for embedding models
- **Dual Crawler Support** - Playwright for JavaScript SPAs, Cheerio for static HTML (faster)
- **Link Graph** - Extracts internal link relationships for building knowledge graphs
- **Content Hashing** - SHA-256 hashes for detecting content changes
- **Smart Content Extraction** - Automatically identifies main content and removes navigation/noise
- **URL Normalization** - Handles query params, trailing slashes, and tracking parameters

### Output Datasets

The crawler generates multiple dataset types (identified by `_datasetType`):

#### Pages (`_datasetType: 'pages'`)

Full page data including:

- `url`, `normalizedUrl`, `canonicalUrl`
- `title`, `h1`, `language`
- `text` - Plain text content
- `markdown` - Converted markdown
- `excerpt` - First 300 characters
- `depth` - Crawl depth from start URL
- `referrers` - URLs that linked to this page
- `outgoingInternalLinks`, `outgoingExternalLinks`
- `contentHash` - SHA-256 hash of markdown content
- `fetchedAt` - ISO timestamp

#### Chunks (`_datasetType: 'chunks'`)

RAG-ready content chunks:

- `chunkId` - Stable unique identifier
- `url`, `normalizedUrl`
- `chunkIndex` - Position in document
- `headingPath` - Array of parent headings (e.g., `["Getting Started", "Installation"]`)
- `markdown`, `text` - Chunk content
- `charStart`, `charEnd` - Character positions in original document
- `chunkHash` - Hash of chunk content
- `pageContentHash` - Hash of parent page
- `tokenEstimate` - Approximate token count

#### Edges (`_datasetType: 'edges'`)

Internal link graph:

- `from` - Source URL (normalized)
- `to` - Target URL (normalized)
- `type` - Link type (`a[href]`)
- `anchorText` - Link text

#### Issues (`_datasetType: 'issues'`)

Crawl errors and warnings:

- `type` - Error type
- `url` - Affected URL
- `message` - Error message
- `severity` - Error severity level

### Input Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `domain` | string | *required* | Domain to crawl (e.g., `https://docs.example.com`) |
| `startUrls` | array | `[]` | Override start URLs (optional) |
| `maxPages` | integer | `200` | Maximum pages to crawl (1-10,000) |
| `maxDepth` | integer | `4` | Maximum crawl depth (1-10) |
| `makeRagReady` | boolean | `true` | Generate RAG-ready chunks |
| `mode` | string | `"docs"` | Extraction mode: `docs`, `article`, `generic` |
| `output` | string | `"all"` | Output: `all`, `pagesOnly`, `chunksOnly`, `edgesOnly` |
| `crawlerType` | string | `"playwright"` | Engine: `playwright` (for SPAs) or `cheerio` (for static) |
| `includeSubdomains` | boolean | `false` | Also crawl subdomains |
| `respectRobotsTxt` | boolean | `true` | Follow robots.txt rules |
| `removeSelectors` | array | `["nav", "aside", ...]` | CSS selectors to remove |
| `allowPatterns` | array | `[]` | Regex patterns for URLs to include |
| `denyPatterns` | array | `[".*utm_.*", ...]` | Regex patterns for URLs to exclude |
| `stripQueryParams` | boolean | `true` | Remove query parameters from URLs |
| `chunkTargetChars` | integer | `2500` | Target chunk size (500-10,000) |
| `chunkMaxChars` | integer | `4500` | Maximum chunk size (1,000-20,000) |
| `minChunkChars` | integer | `400` | Minimum chunk size (100-2,000) |
| `proxyConfiguration` | object | - | Apify proxy settings |

### Example Input

```json
{
    "domain": "https://docs.convex.dev",
    "maxPages": 500,
    "maxDepth": 5,
    "makeRagReady": true,
    "mode": "docs",
    "output": "all",
    "crawlerType": "playwright",
    "chunkTargetChars": 2500,
    "chunkMaxChars": 4500
}
```

### Crawler Types

#### Playwright (default)

- Best for: JavaScript SPAs, React/Vue/Next.js documentation sites
- Waits for `domcontentloaded` + content selectors for fast, reliable extraction
- Slower but handles dynamic content
- Timeout: 90 seconds per page (45s navigation)

#### Cheerio

- Best for: Static HTML sites, traditional documentation
- Much faster (no browser required)
- Lower resource usage
- Timeout: 45 seconds per page

### Content Extraction

The crawler uses smart selectors to find main content:

**Docs mode** tries (in order):

1. `main`, `article`, `[role="main"]`
2. `.content`, `.markdown`, `.prose`
3. `.theme-doc-markdown`, `.md-content`, `.docs-content`
4. Falls back to `body`

Automatically removes noise elements:

- `nav`, `aside`, `header`, `footer`
- `.toc`, `.sidebar`, `.navigation`, `.menu`
- Any custom selectors you specify

### Chunking Strategy

Content is split into chunks based on:

1. **Heading boundaries** - New chunks at `#`, `##`, `###`, `####` headings
2. **Target size** - Aims for ~2,500 characters per chunk
3. **Max size** - Hard limit at 4,500 characters
4. **Min size** - Avoids tiny chunks under 400 characters
5. **Paragraph preservation** - Splits at paragraph boundaries when possible
6. **Sentence preservation** - Falls back to sentence/word boundaries for very long paragraphs

Each chunk includes its `headingPath` for context, making it ideal for RAG systems.

### Local Development

```bash
## Install dependencies
npm install

## Run locally
apify run

## Run with input
apify run --input='{"domain": "https://docs.example.com"}'

## Deploy to Apify
apify push
```

### Technical Notes

- **9MB Limit**: Apify dataset items have a ~9MB limit. Pages exceeding this are automatically truncated (with `truncated: true` flag).
- **URL Normalization**: URLs are normalized (HTTPS, no trailing slashes, tracking params stripped) for deduplication.
- **Content Hashes**: Use `contentHash` and `chunkHash` fields to detect content changes between crawls.
- **Stable Chunk IDs**: `chunkId` is deterministic based on URL, position, and content - same content = same ID.

### Dependencies

- [Apify SDK](https://docs.apify.com/sdk/js/) - Actor framework
- [Crawlee](https://crawlee.dev) - Web scraping library
- [Playwright](https://playwright.dev) - Browser automation
- [Turndown](https://github.com/mixmark-io/turndown) - HTML to Markdown conversion

### License

ISC

# Actor input Schema

## `domain` (type: `string`):

Domain to crawl (e.g., https://docs.example.com)

## `startUrls` (type: `array`):

URLs to start with (optional override for domain)

## `maxPages` (type: `integer`):

Maximum number of pages to crawl

## `maxDepth` (type: `integer`):

Maximum crawl depth from start URLs

## `makeRagReady` (type: `boolean`):

Generate RAG-ready chunks with stable IDs and hashes

## `mode` (type: `string`):

Extraction strategy optimized for different site types

## `output` (type: `string`):

Which datasets to generate

## `includeSubdomains` (type: `boolean`):

Also crawl subdomains of the main domain

## `respectRobotsTxt` (type: `boolean`):

Follow robots.txt rules for crawling

## `crawlerType` (type: `string`):

Crawling engine to use. Use 'playwright' for SPAs/JavaScript-heavy sites, 'cheerio' for static HTML sites (faster)

## `removeSelectors` (type: `array`):

CSS selectors to remove from content

## `allowPatterns` (type: `array`):

Regex patterns for URLs to include

## `denyPatterns` (type: `array`):

Regex patterns for URLs to exclude

## `stripQueryParams` (type: `boolean`):

Remove query parameters from URLs

## `chunkTargetChars` (type: `integer`):

Target number of characters per chunk

## `chunkMaxChars` (type: `integer`):

Maximum number of characters per chunk

## `minChunkChars` (type: `integer`):

Minimum number of characters per chunk

## `proxyConfiguration` (type: `object`):

Proxy settings for crawling

## Actor input object example

```json
{
  "domain": "https://docs.convex.dev",
  "startUrls": [],
  "maxPages": 20,
  "maxDepth": 2,
  "makeRagReady": true,
  "mode": "docs",
  "output": "all",
  "includeSubdomains": false,
  "respectRobotsTxt": true,
  "crawlerType": "playwright",
  "removeSelectors": [
    "nav",
    "aside",
    "header",
    "footer",
    ".toc",
    "#TableOfContents",
    ".table-of-contents"
  ],
  "allowPatterns": [],
  "denyPatterns": [
    ".*(\\?|&)utm_.*",
    ".*(\\?|&)(ref|source|campaign)=.*"
  ],
  "stripQueryParams": true,
  "chunkTargetChars": 2500,
  "chunkMaxChars": 4500,
  "minChunkChars": 400
}
```

# Actor output Schema

## `pagesDataset` (type: `string`):

No description

## `chunksDataset` (type: `string`):

No description

## `edgesDataset` (type: `string`):

No description

## `issuesDataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domain": "https://docs.convex.dev",
    "startUrls": [],
    "maxPages": 20,
    "maxDepth": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("devwithbobby/docs-markdown-rag-ready-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domain": "https://docs.convex.dev",
    "startUrls": [],
    "maxPages": 20,
    "maxDepth": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("devwithbobby/docs-markdown-rag-ready-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domain": "https://docs.convex.dev",
  "startUrls": [],
  "maxPages": 20,
  "maxDepth": 2
}' |
apify call devwithbobby/docs-markdown-rag-ready-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devwithbobby/docs-markdown-rag-ready-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nTb37eddEtg2Aaq7g/builds/Clln9ZvQdWeLuZ2Tc/openapi.json