# RagChunk: Semantic RAG Splitter & Markdown Cleaner (`flaretool/rag-semantic-chunker`) Actor

Extract, clean, and semantically split documentation URLs into token-safe chunks for Vector DBs. Built-in HTML boilerplate stripping saves up to 40% on LLM token costs.

- **URL**: https://apify.com/flaretool/rag-semantic-chunker.md
- **Developed by:** [FlareTool](https://apify.com/flaretool) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## ⚡ RAG Semantic Document Chunker & HTML Cleaner

> **The only RAG preprocessing tool that thinks like a developer and charges like a cloud function.**

Transform any public documentation URL into clean, token-precise, context-rich Markdown chunks, ready to be dropped directly into OpenAI, Pinecone, Weaviate, Chroma, LangChain, or any LLM pipeline. No headless browsers. No bloated DOM trees. No wasted tokens.

***

### 🤔 What Problem Does This Solve?

If you have ever tried to build a **RAG (Retrieval-Augmented Generation)** pipeline on real-world documentation, you already know the pain:

- You scrape a documentation page and your markdown output is **60% navigation links, sidebar menus, cookie banners, and footer noise**, none of which helps your LLM.
- You try to chunk the raw text but end up **slicing code blocks in half**, breaking tables mid-row, or losing which section a chunk belongs to.
- You burn **$15 to $40 in OpenAI embedding tokens per document ingestion run** because you are vectorizing sidebar garbage instead of actual content.

**This Actor solves all three problems in a single API call.**

***

### 🚀 The Differentiator: Up to 40% Token Savings

Standard scrapers ingest entire DOM trees, wasting thousands of dollars on navigation menus, footers, and tracking scripts. This engine runs an advanced **Cheerio-based selector priority extraction cascade** and **link-density evaluation engine** to strip layout noise before any chunking occurs.

**Live Validation Benchmark (Node.js API Reference Page):**

| Metric | Value |
|---|---|
| Raw Markdown Volume | `395,669` characters |
| Post-Processed Clean Markdown | `333,964` characters |
| **Net Token Savings** | **61,705 characters (approximately 15,000 to 20,000 wasted tokens eliminated)** |

That is real money saved on every single document you ingest.

***

### ✅ Who Is This For?

This tool is built for:

- **AI Engineers** building RAG pipelines on top of GPT-4, Claude, Gemini, or open-source LLMs.
- **Developer Tool Teams** ingesting API reference documentation into vector databases.
- **Startups & SaaS companies** building documentation search, AI assistants, or knowledge base copilots.
- **Data engineers** automating content ingestion workflows into Pinecone, Weaviate, Chroma, or pgvector.

If you are feeding documentation into a vector database, this is the highest-quality, lowest-cost preprocessing layer available on the market.

***

### ✨ Core Engine Features

#### 🧹 Aggressive Boilerplate Sanitization

Removes navigation menus, sidebars, footers, tracking pixels, cookie banners, and advertisement containers using a multi-layer Cheerio-based DOM cascade before any conversion to Markdown occurs. Unlike raw scrapers that hand you the entire DOM, we hand you only the signal.

#### 🛡️ Code Block Integrity Protection

The chunker tracks open and closed Markdown code fences (` ``` `) in real-time. If a split would slice a code block in half, it automatically **postpones the split** until the fence closes. Your LLM will always receive complete, syntactically valid code samples.

#### 📊 Markdown Table Integrity Protection

Tables carry structured data (such as parameter schemas, API response mappings, and comparison grids). Our engine detects table boundaries and applies a configurable `table_hard_multiplier` budget to **keep entire tables within a single chunk**, preventing broken header rows from stranding orphaned data rows in the next chunk.

#### 🏷️ Multi-Header Context Tracking

Every emitted chunk includes a `header_contexts` array listing **all active parent headings** within the sliding window boundary. When your RAG retriever returns a chunk about `reduce()`, it will know whether that chunk came from `Parameters`, `Description`, or `Edge cases`. This eliminates the context-drift problem permanently.

#### 🪙 Exact BPE Token Tracking

Uses the `cl100k_base` Byte Pair Encoding tokenizer, which is the exact same tokenizer used internally by GPT-4 and GPT-4o, to count tokens with 100% accuracy. No approximations. No character-based guesses. Every chunk respects the token ceiling you configure.

#### ⛓️ Cascading Line Segmentation

Handles dense layouts (such as blog indexes, link lists, or documentation menus) that render thousands of characters onto a single line. The parser automatically cascades down through sentence boundaries, list boundaries, semicolon/comma clauses, and word limits to segment long lines below `max_tokens * 0.8`. This prevents runaway chunk overlaps and ensures strict token bounds.

#### 🔁 Configurable Sliding Overlap Window

Carries a configurable number of trailing tokens from the previous chunk into the next, ensuring that semantic ideas which straddle chunk boundaries are never completely lost during vector retrieval.

***

### 📥 Input Configuration

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | `string` | **required** | The exact public URL to scrape, extract, and chunk. |
| `max_tokens` | `integer` | `500` | Max BPE tokens per chunk (50–2000). |
| `overlap` | `integer` | `50` | Trailing overlap tokens between consecutive chunks (0–500). |
| `contentSelector` | `string` | *none* | Optional CSS selector to target specific content containers (e.g. `main`, `article`, `#content`). |
| `min_token_floor` | `integer` | `30` | Minimum token count to emit a chunk. Discards tiny non-semantic fragments (5–500). |
| `prose_hard_multiplier` | `number` | `1.2` | Hard token budget multiplier for prose segments before splitting (1.0–2.0). |
| `table_hard_multiplier` | `number` | `3.0` | Hard token budget multiplier inside tables to prevent row fragmentation (1.5–5.0). |

#### Example Input

```json
{
  "url": "https://docs.stripe.com/api/payment_intents",
  "max_tokens": 400,
  "overlap": 40,
  "contentSelector": "main"
}
```

***

### 📤 Output Payload

Each Actor run returns a structured JSON dataset where **every item is one semantically-clean chunk**:

```json
[
  {
    "chunk_index": 0,
    "token_count": 387,
    "header_contexts": [
      "Payment Intents API",
      "Create a PaymentIntent"
    ],
    "content": "## Create a PaymentIntent\n\nCreates a PaymentIntent object.\n\n### Parameters\n\n| Parameter | Type | Description |\n|---|---|---|\n| `amount` | integer | Amount intended to be collected..."
  },
  {
    "chunk_index": 1,
    "token_count": 412,
    "header_contexts": [
      "Create a PaymentIntent",
      "Returns"
    ],
    "content": "Returns a PaymentIntent object if the call succeeded. Returns an error if something goes wrong..."
  }
]
```

#### Output Fields Explained:

- **`chunk_index`**: Sequential position of the chunk within the document.
- **`token_count`**: Exact BPE token count of this chunk's content.
- **`header_contexts`**: All active section headings that were open when this chunk was emitted.
- **`content`**: The clean, pristine GFM Markdown content ready for embedding.

***

### 🧪 Validated Against 50+ Real-World Documentation Sites

We stress-tested this engine against a comprehensive 50-URL live matrix covering Wikipedia, MDN, technical docs (React, AWS, Stripe), personal blogs (Paul Graham), forums (Stack Overflow), and company landing pages (Hugging Face, Google DeepMind, Microsoft Research). It achieved:

- **100% execution success** across all accessible layouts.
- **0 broken code blocks or table rows** (100% formatting preservation).
- **Runaway overlap prevention** on complex dense lists.

#### Live Metrics Highlight:

| Target URL / Layout Type | Clean Markdown | Chunks Generated | Min Tokens | Max Tokens | Avg Tokens | Code Block Integrity |
|---|---|---|---|---|---|---|
| **Wikipedia:** [Retrieval-augmented generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) | 25,214 chars | 33 | 101 | 480 | 353 | ✅ PASS |
| **Wikipedia:** [Transformer Architecture](https://en.wikipedia.org/wiki/Transformer_\(deep_learning_architecture\)) | 184,897 chars | 281 | 117 | 679 | 402 | ✅ PASS |
| **MDN:** [Array.prototype.reduce()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce) | 17,763 chars | 14 | 328 | 510 | 399 | ✅ PASS |
| **Technical Doc:** [React useEffect Reference](https://react.dev/reference/react/useEffect) | 44,352 chars | 36 | 217 | 501 | 367 | ✅ PASS |
| **Personal Blog:** [How to Do Great Work (Paul Graham)](https://paulgraham.com/greatwork.html) | 69,016 chars | 45 | 65 | 472 | 408 | ✅ PASS |
| **Stripe API:** [PaymentIntents Reference](https://stripe.com/docs/api/payment_intents/create) | 21,182 chars | 27 | 70 | 689 | 401 | ✅ PASS |
| **Forum:** [Stack Overflow Sorted Array Q\&A](https://stackoverflow.com/questions/11227809/) | 140,564 chars | 141 | 55 | 584 | 346 | ✅ PASS |
| **Company Blog:** [Astro Blog (Astro 4 Release)](https://astro.build/blog/astro-4/) | 11,871 chars | 8 | 312 | 437 | 380 | ✅ PASS |
| **Aggregator:** [Hugging Face Blog Homepage](https://huggingface.co/blog) | 11,698 chars | 16 | 183 | 579 | 428 | ✅ PASS |

***

### ⚙️ Performance Benchmarks

| Metric | Value |
|---|---|
| Median Latency (p50) | ~497ms end-to-end |
| Scraping Engine | Lightweight DOM parsing (no headless browser) |
| Token Tracking | `cl100k_base` BPE (GPT-4/GPT-4o compatible) |
| Memory Footprint | Resource-isolated, no persistent storage |
| Pricing | $2.00 / 1,000 chunks (pay-per-result, not per-run) |

***

### ❓ Frequently Asked Questions

**Q: What types of pages does this work best on?**
A: It excels on developer documentation, API reference pages, technical guides, blog articles, and knowledge base content. Any structured, text-rich public web page.

**Q: Does it work on sites protected by Cloudflare or other bot protections?**
A: The engine uses a multi-tier fetch pipeline. If direct access is blocked, it automatically falls back to a secondary content extraction layer to maximize success rates.

**Q: How is this priced?**
A: You pay **$2.00 per 1,000 chunks** generated. A typical documentation page generates 8–15 chunks, meaning a full page costs less than $0.03. There are no monthly minimums, no seat fees, and no API key commitments.

**Q: Can I use this to ingest private/internal documentation?**
A: This Actor only fetches publicly accessible URLs. It cannot bypass authentication or access pages behind a login wall.

**Q: Which vector databases does the output work with?**
A: The output is plain JSON with string content fields. It is compatible with every major vector database including Pinecone, Weaviate, Chroma, Qdrant, pgvector, Milvus, and any LangChain / LlamaIndex embedding pipeline.

**Q: Does it preserve code examples from documentation?**
A: Yes. This is one of its core differentiators. The engine specifically detects and protects Markdown code fences, refusing to split a chunk mid-code-block regardless of token budget.

***

### Legal & Licensing

Copyright © 2026. All Rights Reserved. Private Proprietary Software. Usage is strictly governed by the [Terms of Service](https://chunk.flaretool.com/terms) and [Privacy Policy](https://chunk.flaretool.com/privacy).

# Actor input Schema

## `url` (type: `string`):

The exact URL of the webpage or documentation link you want to scrape, extract, and chunk.

## `max_tokens` (type: `integer`):

The max token allocation target for each sliding window slice using the cl100k_base (GPT-4) tokenizer footprint.

## `overlap` (type: `integer`):

The number of trailing tokens carried over from the previous chunk to maintain semantic context boundaries.

## `contentSelector` (type: `string`):

Optional CSS selector prioritizing specific wrapper elements for main content extraction (e.g. 'main', 'article', '#content').

## `min_token_floor` (type: `integer`):

The minimum number of tokens required to emit a chunk. Discards tiny, non-semantic fragments.

## `prose_hard_multiplier` (type: `number`):

Hard token limit multiplier for prose segments before splitting (1.0 - 2.0).

## `table_hard_multiplier` (type: `number`):

Hard token limit multiplier inside tables to prevent data fragmentation (1.5 - 5.0).

## Actor input object example

```json
{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "max_tokens": 400,
  "overlap": 40,
  "contentSelector": "",
  "min_token_floor": 30,
  "prose_hard_multiplier": 1.2,
  "table_hard_multiplier": 3
}
```

# Actor output Schema

## `results` (type: `string`):

An array of structured, tokenized Markdown segments.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "contentSelector": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("flaretool/rag-semantic-chunker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "contentSelector": "",
}

# Run the Actor and wait for it to finish
run = client.actor("flaretool/rag-semantic-chunker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "contentSelector": ""
}' |
apify call flaretool/rag-semantic-chunker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,flaretool/rag-semantic-chunker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lhCFm2ZHu5zVW8fIj/builds/Vm9lljubGAHIFf8sb/openapi.json