# AI Web Crawler (`hounderd/ai-web-crawler`) Actor

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

- **URL**: https://apify.com/hounderd/ai-web-crawler.md
- **Developed by:** [Hounderd](https://apify.com/hounderd) (community)
- **Categories:** AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 99 total users, 7 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 page extracteds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Web Crawler — LLM-Ready Content Extractor

Turn any website into clean, structured content for AI pipelines, RAG systems, and data workflows. Uses a real browser with stealth rendering to bypass Cloudflare, anti-bot systems, and JavaScript-heavy pages that basic scrapers can't touch.

***

### 🚀 What does this do?

This Actor crawls websites and extracts page content in formats built for LLMs:

- **Clean Markdown** — boilerplate stripped, ready to feed into any AI model
- **AI-Optimized Markdown** — noise removed via intelligent content filtering, maximizes signal-to-noise for RAG and embeddings
- **Full-site crawling** — follow links automatically with BFS or DFS traversal
- **Stealth browser extraction** — Camoufox-based rendering improves success on Cloudflare-challenged, anti-bot, and JavaScript-heavy pages
- **Structured metadata** — title, description, Open Graph, author, language per page
- **Token estimation** — word count and estimated token count for every page

Runs via Apify API, webhooks, and schedules — no code required to get started.

***

### 📦 Output Data

| Field | Description |
|-------|-------------|
| **url** | The crawled page URL |
| **title** | Page `<title>` tag |
| **statusCode** | HTTP response status code |
| **markdown** | Full page content as clean Markdown |
| **fitMarkdown** | AI-optimized Markdown with boilerplate filtered out |
| **rawHtml** | Original HTML (optional) |
| **cleanedHtml** | HTML with boilerplate removed (optional) |
| **screenshot** | Base64 PNG screenshot of the page (optional) |
| **wordCount** | Number of words in the extracted content |
| **estimatedTokens** | Rough token count (~4 chars/token) |
| **contentLength** | Character count of extracted content |
| **metadata.description** | Meta description |
| **metadata.keywords** | Meta keywords |
| **metadata.author** | Page author |
| **metadata.language** | Page language |
| **metadata.ogTitle** | Open Graph title |
| **metadata.ogDescription** | Open Graph description |
| **metadata.ogImage** | Open Graph image URL |

#### Example output

```json
{
  "url": "https://docs.example.com/getting-started",
  "title": "Getting Started — Example Docs",
  "statusCode": 200,
  "markdown": "# Getting Started\n\nWelcome to Example...",
  "fitMarkdown": "# Getting Started\n\nWelcome to Example...",
  "wordCount": 843,
  "estimatedTokens": 1124,
  "contentLength": 4498,
  "metadata": {
    "description": "Learn how to get started with Example in minutes.",
    "keywords": "getting started, tutorial, example",
    "author": "Example Team",
    "language": "en",
    "ogTitle": "Getting Started — Example Docs",
    "ogDescription": "Learn how to get started with Example in minutes.",
    "ogImage": "https://docs.example.com/og-getting-started.png"
  }
}
```

***

### 💡 Use Cases

- **RAG Pipelines** — Ingest documentation, blogs, or knowledge bases into vector stores
- **AI Research** — Gather clean text from multiple pages for analysis or summarization
- **Documentation Scraping** — Extract entire doc sites into Markdown for offline use or fine-tuning
- **Competitive Intelligence** — Monitor competitor pages and detect content changes
- **Content Migration** — Convert any website to Markdown for import into Notion, Obsidian, or CMS tools
- **LLM Context Prep** — Feed live web content into AI agents and chatbots

***

### ⚙️ Options

| Option | Description |
|--------|-------------|
| **startUrls** | One or more URLs to crawl |
| **crawlMode** | `single` (start URLs only), `bfs` (breadth-first), or `dfs` (depth-first) |
| **maxCrawlDepth** | How many link-hops deep to follow from start URLs (BFS/DFS only) |
| **maxCrawlPages** | Maximum total pages to crawl per run |
| **sameDomainOnly** | Only follow links within the same domain (default: on) |
| **includeUrlPatterns** | Regex patterns — only follow URLs that match |
| **excludeUrlPatterns** | Regex patterns — skip URLs that match (e.g. `/login`, `\.pdf$`) |
| **outputFormats** | Choose any combination: `markdown`, `fitMarkdown`, `rawHtml`, `cleanedHtml`, `screenshot` |
| **cssSelector** | Restrict extraction to a specific part of the page (e.g. `article`, `main`, `#content`) |
| **excludeSelectors** | CSS selectors for elements to strip before extraction (e.g. `nav`, `.sidebar`) |
| **waitForSelector** | Wait for a CSS selector to appear before extracting — useful for JS-rendered pages |
| **waitForTimeout** | Extra wait time in ms after page load (for lazy-loaded content) |
| **executeJavaScript** | Custom JS to run on each page before extraction (dismiss popups, click "show more", etc.) |
| **scrollToBottom** | Scroll the full page to trigger lazy-loaded and infinite-scroll content |
| **includeLinks** | Preserve hyperlinks in Markdown output (default: on) |
| **includeImages** | Include image references in Markdown output (default: on) |
| **includeMetadata** | Extract and include page metadata block (default: on) |
| **maxConcurrency** | Pages to crawl in parallel in standard mode (default: 5, max: 20). Stealth mode crawls sequentially for reliability. |
| **requestTimeout** | Max total seconds to spend on a page before giving up. In stealth mode this budget includes page load, challenge waits, selector waits, and retries (default: 30) |
| **stealthMode** | Enable stealth browser rendering to bypass bot detection (default: on, recommended) |
| **proxyConfiguration** | Optional proxy settings — Residential proxies are recommended for protected sites, but not required for ordinary public pages |

***

### 🛡️ Anti-Bot & Cloudflare Bypass

Most scrapers fail on modern websites because they're caught by bot detection at the browser and IP reputation layers. This Actor uses a hardened stealth browser path plus proxy support to reduce fingerprint-based detection and improve extraction success on tougher targets.

For the best results on Cloudflare-protected or heavily guarded sites:

1. **Enable Stealth Mode** (default: on) — uses the Camoufox-based path for lower-friction browser fingerprinting
2. **Use Residential Proxies for guarded targets** — datacenter IPs are blocked much more aggressively by systems like Cloudflare and Akamai, but many ordinary public sites do not need proxy spend at all

These settings materially improve compatibility with sites protected by systems like Cloudflare, Akamai, DataDome, and PerimeterX, but some sites may still challenge or block requests depending on IP reputation and challenge type.

***

### 🔗 API Usage

Trigger a crawl via the Apify API:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/hounderd~ai-web-crawler/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://docs.example.com"}],
    "crawlMode": "bfs",
    "maxCrawlDepth": 2,
    "maxCrawlPages": 50,
    "outputFormats": ["markdown", "fitMarkdown"]
  }'
```

Results are available in the run's default dataset once the status is `SUCCEEDED`.

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs to crawl.

## `crawlMode` (type: `string`):

How to discover and follow links. 'single' only crawls the start URLs. 'bfs' explores level by level (breadth-first). 'dfs' goes deep before wide (depth-first).

## `maxCrawlDepth` (type: `integer`):

Maximum link-following depth from start URLs. Only used when crawl mode is BFS or DFS.

## `maxCrawlPages` (type: `integer`):

Maximum total number of pages to crawl.

## `sameDomainOnly` (type: `boolean`):

Only follow links within the same domain as the start URL.

## `includeUrlPatterns` (type: `array`):

Only follow URLs matching these regex patterns. Leave empty to follow all URLs.

## `excludeUrlPatterns` (type: `array`):

Skip URLs matching these regex patterns (e.g., '/login', '/signup', '.pdf$').

## `outputFormats` (type: `array`):

Which content formats to include in the output. 'markdown' is clean markdown. 'fitMarkdown' is AI-optimized with noise removed via BM25 filtering. 'rawHtml' is the original HTML. 'cleanedHtml' has boilerplate removed. 'screenshot' is a base64 PNG.

## `cssSelector` (type: `string`):

Restrict extraction to a specific part of the page (e.g., 'article', 'main', '#content', '.post-body'). Leave empty for full page.

## `excludeSelectors` (type: `array`):

CSS selectors for elements to remove before extraction (e.g., 'nav', 'footer', '.sidebar', '.cookie-banner').

## `waitForSelector` (type: `string`):

Wait for this CSS selector to appear before extracting content. Useful for JS-rendered pages.

## `waitForTimeout` (type: `integer`):

Extra time to wait after page load (milliseconds). Useful for lazy-loaded content.

## `executeJavaScript` (type: `string`):

Custom JavaScript to execute on each page before extraction. Useful for clicking 'show more' buttons, dismissing popups, etc.

## `scrollToBottom` (type: `boolean`):

Scroll the full page before extraction. Triggers lazy-loaded images and infinite scroll content.

## `includeLinks` (type: `boolean`):

Preserve hyperlinks in the markdown output. Disable for cleaner text-only output.

## `includeImages` (type: `boolean`):

Include image references in markdown output.

## `includeMetadata` (type: `boolean`):

Extract and include page metadata (description, keywords, author, language, Open Graph data).

## `maxConcurrency` (type: `integer`):

Maximum number of pages to crawl in parallel in standard mode. Stealth mode crawls sequentially for reliability.

## `requestTimeout` (type: `integer`):

Maximum total time to spend on one page before giving up. In stealth mode this budget includes page load, challenge waits, selector waits, and retries.

## `stealthMode` (type: `boolean`):

Enable the Camoufox-based stealth browser path to reduce browser-fingerprint detection on protected sites. Recommended for guarded targets, but some sites may still challenge or block depending on IP reputation and challenge type.

## `proxyConfiguration` (type: `object`):

Optional proxy settings. Residential proxies are recommended for sites with anti-bot protection (Cloudflare, Akamai, DataDome), but they are not required for ordinary public pages.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/getting-started"
    }
  ],
  "crawlMode": "single",
  "maxCrawlDepth": 1,
  "maxCrawlPages": 10,
  "sameDomainOnly": true,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "outputFormats": [
    "markdown",
    "fitMarkdown"
  ],
  "excludeSelectors": [
    "nav",
    "footer",
    "header",
    ".cookie-banner",
    ".consent-popup",
    "#cookie-notice"
  ],
  "waitForTimeout": 0,
  "scrollToBottom": false,
  "includeLinks": true,
  "includeImages": true,
  "includeMetadata": true,
  "maxConcurrency": 5,
  "requestTimeout": 30,
  "stealthMode": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/getting-started"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hounderd/ai-web-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/getting-started" }] }

# Run the Actor and wait for it to finish
run = client.actor("hounderd/ai-web-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/getting-started"
    }
  ]
}' |
apify call hounderd/ai-web-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hounderd/ai-web-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rXji6VhvfBVbIzWdb/builds/xHhcyhbEYH9sUNl5h/openapi.json