# Website Content Crawler — Text, Markdown & AI Chunks (`agency-shift/web-content-crawler`) Actor Extract text or clean Markdown from public HTML websites. Create source-linked sections for AI knowledge bases while keeping one row per page. Preserve headings, code and tables; inspect a crawl report for failed URLs and truncated content. No JavaScript rendering. - **URL**: https://apify.com/agency-shift/web-content-crawler.md - **Developed by:** [Mako](https://apify.com/agency-shift) (community) - **Categories:** Developer tools, AI - **Features:** Limited permissions - **Stats:** 60 total users, 8 monthly users, 98.4% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.035 / actor start This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### Turn website documentation into a source-linked knowledge base Extract **plain text, clean Markdown and source-linked sections** from public HTML websites. Use the result for an AI knowledge base, document search or a content audit. Headings, links, code examples and tables stay with their source URLs so downstream results can cite the original page. This is a content preparation tool. It does not call an AI model, create embeddings, answer questions or render JavaScript. Existing API inputs keep the original text output; Markdown and chunks are optional. All modes keep **one diagnostic dataset row per fetched page**, with chunks nested inside that row. A saved row does not itself prove useful content: check `contentUsable`, `contentStatus`, `contentWarnings` and the crawl report before consuming it. ### Inspect real output before your first run **[Preview a real two-page extraction](https://makorev.com/apis/website-research)** with Markdown, JSON and source-linked chunks. The sample is freely readable and downloadable without signing in. It contains Mako-owned public pages captured on October 2, 2026: **2 pages, 33 chunks, zero failed requests and zero Markdown-truncated pages**. This is dated test evidence, not a promise that every website can be extracted. **[Download the n8n website-research starter](https://github.com/valdeircs/scraper-fleet/tree/main/examples/website-research)** to run the same bounded workflow with your own Apify account. It starts manually, applies a $0.10 Actor charge ceiling and 180-second timeout, retrieves both the crawl report and dataset, and checks for missing or truncated content before preparing a source packet. The ceiling is not the price: at 512 MB the current Free-plan startup event is $0.05. No paid AI model, CRM connection or community node is required. The exported workflow contains no credentials and is inactive by default. If the supplied requests fail, the report is missing, or content is incomplete, the workflow stops instead of presenting that output as a completed research packet. Extracted page content remains untrusted source material; it is not an instruction to an AI assistant. Inspect it before connecting any downstream action. **Prefer an AI assistant?** Review the optional [Mako website-research skill and connection instructions](https://github.com/valdeircs/scraper-fleet/tree/main/skills). It uses your own authenticated Apify MCP connection and requires a run budget; downloading it does not install or authorize a connection. ### Get your first three pages **[Open the three-page example](https://apify.com/agency-shift/web-content-crawler/examples/extract-documentation-text-for-ai)**, choose **Text + Markdown + source-linked chunks**, or use the input below. Use 512 MB memory, keep link following and proxy use off, and start with three public pages. ```json { "startUrls": [ "https://docs.apify.com/actors", "https://docs.apify.com/actors/development/actor-definition", "https://docs.apify.com/actors/development/actor-definition/output-schema" ], "maxPages": 3, "followLinks": false, "maxTextLength": 20000, "useProxy": false, "exportFormat": "markdown-and-chunks", "maxMarkdownLength": 100000, "chunkMaxCharacters": 2000 } ``` After the run: 1. Open **Crawl report: coverage and failures**. Check `outcome` and `usablePageCount`, then review failed/unprocessed URLs, `contentIssues` and `linkDiscoveryFailures`. Keep dataset rows with `contentUsable: true` for downstream ingestion; review flagged rows separately. 2. Open **Crawled pages → Markdown and AI chunks**. Review the Markdown and check `markdownTruncated`, `chunkCount` and `oversizedChunkCount`. 3. Download the dataset as **JSON** to preserve nested chunks. CSV is useful for page-level fields but is not the recommended knowledge-base interchange format. 4. Import the chunks into your destination, retaining `sourceUrl`, `sectionUrl`, `title` and `headingPath` beside the content. The SQLite example below demonstrates retrieval with source citations without a paid model. At 512 MB, the current Free-plan Actor charge is **$0.05 per run**, with no per-page, per-chunk or AI-model fee. Other tiers and storage qualifications appear below. Changing export format does not add an Actor billing event. ### What you receive Every mode adds `contentStatus`, `contentUsable`, `contentWarnings`, `contentCharacters`, `contentQualityThreshold` and `extractionMode: "static-html"`. A warning contains a stable `code`, a readable `message` and an `action` to try. Every mode retains `url`, `title`, `description`, `h1`, `text`, `textLength`, `links`, `linkCount`, `crawledAt`, `loadedUrl`, `httpStatusCode`, `textTruncated` and `textLengthBeforeTruncation`. **Markdown mode** adds `markdown`, `markdownLength`, `markdownLengthBeforeTruncation`, `markdownTruncated`, `markdownTruncationBoundary` and `contentSelector`. It selects a semantic main/article or common documentation container, then removes navigation and other noise. This is a heuristic; set the optional `contentSelector` for a site with an unusual layout. The original `text` still uses the previous body-text extraction, so it can contain content absent from Markdown. **Markdown and chunks mode** also adds `chunks`, `chunkCount` and `oversizedChunkCount`. Each nested chunk contains: | Field | Meaning | |---|---| | `id` | Deterministic SHA-256 identifier based on source URL, heading-path occurrence and part index | | `sourceUrl` | Final loaded page URL | | `sectionUrl` | Page URL with a real heading/section anchor when one is present | | `title` / `headingPath` | Page title and the section's heading hierarchy | | `content` | Markdown for this section or part of it | | `contentHash` | SHA-256 hash of this chunk's exact content | | `characterCount` / `oversized` | Content length and whether it exceeds the requested chunk target | IDs remain the same when the source, heading position and chunk boundaries remain the same. Heading edits, inserted duplicate headings, URL changes or rechunking can change IDs. Use `contentHash` to detect exact content changes; these fields do not create a persistent change history for you. Sections start at headings. Blocks are grouped up to the target size; a long paragraph, code block, list or table stays intact and can exceed it. The target is measured in JavaScript string characters, **not model tokens**. Inspect oversized chunks before sending them to a model with a strict token limit. Relative HTTP(S) links and image URLs are resolved against the loaded page URL. Images are linked, not downloaded or analyzed. Simple tables become GitHub-flavored Markdown; complex tables can retain HTML. `maxMarkdownLength` is a separate page cap from `maxTextLength`. When the cap is reached, extraction stops at a block boundary when possible. If even the first block is too large, a Unicode-safe prefix is retained and fenced code is closed; `markdownTruncationBoundary` is then `character`. Character truncation can split inline links or table formatting. Chunks cover only the retained Markdown, never omitted content. ### Know what was collected The **crawl report**, stored as `RUN_SUMMARY` in the run's key-value store, separates returned rows from usable pages, content issues, failed requests, link-discovery failures, unprocessed starting URLs and truncated plain text. Read its `outcome` before using the output: - **`succeeded`:** every saved page passes the content diagnostic and the recorded crawl has no detected collection gap. This does not prove relevance, whole-site coverage or that downloaded HTML contains all rendered content. - **`partial`:** at least one usable page exists, but another page has a content issue, a request/discovery failure, a limit or unresolved coverage. Useful rows stay available. - **`failed`:** no page passes the content diagnostic, even if HTTP requests succeeded and diagnostic rows were saved. The Actor exits with a failure status after saving the report. A fatal error after usable rows were saved is reported as `partial`. The content check assesses the **returned format**: plain text in `text` mode, Markdown in either Markdown mode. Empty content is `empty`; content shorter than **80 trimmed characters** is `low_content`. This threshold is a conservative heuristic: a legitimate very short page may be flagged and is retained for manual review. A short page with an explicit JavaScript-required message becomes `javascript_required`; a detected access-check placeholder becomes `access_challenge`. Script tags alone produce only a possible-rendering warning on an empty/short page, not proof that JavaScript is needed. `usable` does not guarantee that a page contains the information you wanted. `returnedPageCount` counts all saved rows, including diagnostic rows. `usablePageCount` counts rows passing this check; `contentIssuePageCount` and `contentIssues` explain the rest. `successfulRequestCount` refers to successful HTTP extraction/handling, not useful content. For automation or MCP, inspect the report before treating a run as successful, and filter on `contentUsable`. An empty navigation index can be flagged while its discovered article is successfully returned in the same run. AI export modes add an `aiExport` summary containing returned chunk, oversized chunk, empty Markdown page and truncated Markdown page counts. Content truncation is separate from request coverage and does not change `outcome`; always inspect both. Increase the corresponding content cap and rerun when needed. A hard platform termination or storage failure can prevent the final report from being saved. ### Pricing: calculate the Actor charge This Actor charges an **Actor-start event per allocated GB of memory, with a minimum of one event**. There is no per-page or enrichment event. At the recommended **512 MB**, the three-page example uses one start event: | Subscription tier | Actor charge at 512 MB | |---|---:| | Free | $0.050 | | Bronze | $0.045 | | Silver | $0.040 | | Gold, Platinum or Diamond | $0.035 | Rates checked October 2, 2026. The live **Pricing** tab is authoritative. One event covers runs up to 1 GB, so 512 MB is not half-price. At 4 GB, four events cost $0.20 at the listed Free rate. Each new run incurs the startup charge, even if zero usable pages return. Runtime platform usage is included under the current pricing configuration. Post-run dataset operations and retained storage follow Apify's standard charges; downstream AI services are separate. This example keeps proxy use off. The $0.10 maximum-charge setting is a ceiling for Actor events, not the estimated price, a refund promise or a guarantee of complete output. Owner-run operating costs are not the price another customer pays. ### Inputs and larger crawls | Input | Default | Purpose | |---|---:|---| | `startUrls` | Required | Public HTML URLs; the form prefills the documentation example | | `maxPages` | 10; example uses 3 | Crawl request limit, not a promised record count | | `followLinks` | false | Follow links on the same domain | | `maxTextLength` | 20000 | Stored plain-text characters per page | | `useProxy` | false | Optional Apify datacenter proxy | | `exportFormat` | `text` | `text`, `markdown` or `markdown-and-chunks`; the form prefills the AI mode | | `maxMarkdownLength` | 100000 | Markdown page cap: 500–1,000,000 characters | | `chunkMaxCharacters` | 2000 | Chunk target: 300–12,000 characters; blocks can exceed it | | `contentSelector` | Empty | Optional CSS selector for Markdown/chunks, maximum 200 characters | Start with supplied URLs before enabling link following. Crawlee can finish requests already in progress when a page limit is reached; inspect actual counts in the report. Neither exhausting a queue nor reaching a limit establishes whole-site completeness. Only HTTP(S) links are included in Markdown, and an explicit content selector that matches no element fails that page. ### Import into local SQLite search Download **Crawled pages** as a JSON array named `pages.json`. Save the Python script below as `import_sqlite.py`; it uses Python's standard library with SQLite FTS5 and needs no API key. Inspect the crawl report before importing, because a dataset export alone cannot tell you which URLs failed. ```sh python3 import_sqlite.py pages.json knowledge.db --query output --jsonl chunks.jsonl ``` The script imports nested chunks into a searchable local database, prints matching content with its source link, and optionally writes one chunk per JSONL line. Reimporting replaces chunks only for the page URLs in this export, removing stale chunks for those sources while retaining other pages. It rejects truncated Markdown by default. `--allow-truncated` opts into knowingly partial content. This example performs keyword retrieval; embedding generation or AI answers are separate downstream steps.
Copy the tested SQLite import script ```python #!/usr/bin/env python3 """Import a Web Content Crawler JSON export into local SQLite FTS5. No API key required.""" import argparse import hashlib import json from pathlib import Path import sqlite3 import sys def import_pages(pages, connection, allow_truncated=False): if not isinstance(pages, list) or not pages: raise ValueError('Expected a nonempty JSON array exported from the dataset.') documents = {} sources = set() for page in pages: if not isinstance(page, dict) or not isinstance(page.get('chunks'), list): raise ValueError('Each page must contain chunks. Run with exportFormat=markdown-and-chunks.') if page.get('markdownTruncated') and not allow_truncated: raise ValueError('Markdown is truncated; rerun with a larger cap or pass --allow-truncated knowingly.') source = page.get('loadedUrl') or page.get('url') if not isinstance(source, str) or not source.startswith(('http://', 'https://')): raise ValueError('Each page needs its original HTTP(S) source URL.') sources.add(source) for chunk in page['chunks']: if not isinstance(chunk, dict) or chunk.get('sourceUrl') != source: raise ValueError('Chunk source must match its page URL.') for field in ['id', 'sourceUrl', 'sectionUrl', 'content', 'contentHash']: if not isinstance(chunk.get(field), str) or not chunk[field]: raise ValueError('Missing chunk field: ' + field) headings = chunk.get('headingPath') if not isinstance(headings, list) or not all(isinstance(x, str) for x in headings): raise ValueError('headingPath must be an array of strings.') if hashlib.sha256(chunk['content'].encode()).hexdigest() != chunk['contentHash']: raise ValueError('Chunk content does not match its contentHash.') if chunk['id'] in documents and documents[chunk['id']] != chunk: raise ValueError('Conflicting duplicate chunk IDs in the export.') documents[chunk['id']] = chunk with connection: connection.execute('CREATE TABLE IF NOT EXISTS chunks (id TEXT PRIMARY KEY, source_url TEXT NOT NULL, section_url TEXT NOT NULL, title TEXT, heading_path TEXT NOT NULL, content TEXT NOT NULL, content_hash TEXT NOT NULL)') connection.execute('CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(id UNINDEXED, title, headings, content)') # Replace only supplied sources, retaining unrelated previously imported pages. for source in sources: connection.execute('DELETE FROM chunks_fts WHERE id IN (SELECT id FROM chunks WHERE source_url = ?)', (source,)) connection.execute('DELETE FROM chunks WHERE source_url = ?', (source,)) for chunk in documents.values(): title = chunk.get('title') or '' connection.execute('INSERT INTO chunks VALUES (?, ?, ?, ?, ?, ?, ?)', (chunk['id'], chunk['sourceUrl'], chunk['sectionUrl'], title, json.dumps(chunk['headingPath'], ensure_ascii=False), chunk['content'], chunk['contentHash'])) connection.execute('INSERT INTO chunks_fts VALUES (?, ?, ?, ?)', (chunk['id'], title, ' > '.join(chunk['headingPath']), chunk['content'])) return list(documents.values()) def search(connection, query): return connection.execute('SELECT c.title, c.section_url, c.content FROM chunks_fts f JOIN chunks c ON c.id = f.id WHERE chunks_fts MATCH ? ORDER BY bm25(chunks_fts) LIMIT 5', (query,)).fetchall() def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('input_json', type=Path, help='JSON array downloaded from Crawled pages') parser.add_argument('database', type=Path) parser.add_argument('--query', help='SQLite FTS5 search, e.g. output or "actor output"') parser.add_argument('--jsonl', type=Path, help='Optional flat chunk JSONL export for another destination') parser.add_argument('--allow-truncated', action='store_true') args = parser.parse_args() try: pages = json.loads(args.input_json.read_text(encoding='utf-8')) with sqlite3.connect(args.database) as connection: chunks = import_pages(pages, connection, args.allow_truncated) print(f'Imported {len(chunks)} source-linked chunks into {args.database}.') if args.query: results = search(connection, args.query) print(f'{len(results)} matching chunks:') for title, url, content in results: print(f'\n{title}\nSource: {url}\n{content[:600]}') if args.jsonl: args.jsonl.write_text(''.join(json.dumps(chunk, ensure_ascii=False) + '\n' for chunk in chunks), encoding='utf-8') except (OSError, ValueError, sqlite3.Error) as error: print(f'Import failed: {error}', file=sys.stderr) return 1 return 0 if __name__ == '__main__': raise SystemExit(main()) ```
### Coverage, repeat use and support The legacy plain-text extractor removes script, style, noscript, SVG, navigation, header and footer elements, then collapses whitespace. This can remove useful content on some sites. Its legacy `links` field contains up to 100 absolute HTTP(S) links found after cleanup; relative links are excluded and duplicates can remain. **Crawl discovery is separate**: when `followLinks` is on, same-domain links are enqueued from the original HTML before cleanup, including relative navigation, header and footer links. A queueing failure is recorded in `linkDiscoveryFailures`, preserves the extracted row and makes the run partial when usable content exists. The optional Markdown extractor uses a separate cleaned copy and keeps article headings, code whitespace and resolved relative links. Pages requiring JavaScript, login or unsupported access challenges may fail or return incomplete content. Empty/short content and recognized JavaScript-required or access-check placeholders are flagged with suggested next steps; these heuristics cannot detect every blocked or incomplete page. Proxy use does not render JavaScript or guarantee access. This Actor does not extract PDFs, process images, create embeddings or maintain change history. It does not guarantee every page layout is recognized correctly; check representative pages before a large run. Treat scraped content as untrusted source material when passing it to an AI assistant. Save an Apify task to reuse an input, or schedule it for recurring snapshots. Compare exported hashes downstream when you need change detection. For support, open an issue with the run ID and a reproducible public URL, without credentials. # Actor input Schema ## `startUrls` (type: `array`): Public HTML page URLs. Use specific articles or documentation pages for the first test. JavaScript is not rendered; pages with no usable HTML content are flagged in both the dataset and crawl report. ## `maxPages` (type: `integer`): Crawl request budget. Use 3 for the three-page example. Concurrent requests can slightly exceed this number; failures can leave fewer results. Check the crawl report for unprocessed inputs. ## `followLinks` (type: `boolean`): Discover same-domain links from the original HTML, including navigation, header and footer links, before text cleanup. The crawl request budget still applies; this does not guarantee complete website coverage. ## `maxTextLength` (type: `integer`): Maximum text characters stored per page. The output flags truncated text so you can increase this value when needed. ## `useProxy` (type: `boolean`): Optional Apify datacenter proxy. Leave off for the documentation example. It does not render JavaScript or guarantee access to blocked sites. ## `exportFormat` (type: `string`): Always retains legacy text fields and one diagnostic dataset row per fetched page. Omitted API field defaults to text; choose markdown-and-chunks for a knowledge base. Inspect contentUsable and RUN_SUMMARY before using results. No AI model, embeddings or browser rendering is called. ## `maxMarkdownLength` (type: `integer`): Applies only to Markdown/chunks. Stops at a block boundary when possible; a single first block larger than the limit is cut and flagged. Chunks include only this retained Markdown. Check markdownTruncated and the crawl report. ## `chunkMaxCharacters` (type: `integer`): Applies to Markdown and chunks. Groups blocks within each heading section. Code, tables and long paragraphs stay intact, so a chunk can exceed this target; oversized chunks are flagged. This is a character target, not a token limit. ## `contentSelector` (type: `string`): Applies only to Markdown/chunks. Leave blank to choose a semantic main/article or common documentation container, falling back to body. Set a CSS selector for a known page layout; if it matches nothing, that page fails visibly. When multiple elements match, the largest text container is selected. ## Actor input object example ```json { "startUrls": [ "https://docs.apify.com/actors", "https://docs.apify.com/actors/development/actor-definition", "https://docs.apify.com/actors/development/actor-definition/output-schema" ], "maxPages": 3, "followLinks": false, "maxTextLength": 20000, "useProxy": false, "exportFormat": "markdown-and-chunks", "maxMarkdownLength": 100000, "chunkMaxCharacters": 2000, "contentSelector": "" } ``` # Actor output Schema ## `results` (type: `string`): One diagnostic record per fetched page; inspect contentUsable, contentStatus and contentWarnings before using it. Legacy text is retained; optional Markdown and nested source-linked chunks need exportFormat. Export JSON to preserve chunks. ## `coverage` (type: `string`): Read outcome and usablePageCount first. Zero usable content fails the run; useful partial rows stay available. Report includes content issues/actions, failed/unprocessed URLs, discovery failures and truncation. Static HTML only; whole-site completeness is not assessed. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://docs.apify.com/actors", "https://docs.apify.com/actors/development/actor-definition", "https://docs.apify.com/actors/development/actor-definition/output-schema" ], "maxPages": 3, "exportFormat": "markdown-and-chunks" }; // Run the Actor and wait for it to finish const run = await client.actor("agency-shift/web-content-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [ "https://docs.apify.com/actors", "https://docs.apify.com/actors/development/actor-definition", "https://docs.apify.com/actors/development/actor-definition/output-schema", ], "maxPages": 3, "exportFormat": "markdown-and-chunks", } # Run the Actor and wait for it to finish run = client.actor("agency-shift/web-content-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://docs.apify.com/actors", "https://docs.apify.com/actors/development/actor-definition", "https://docs.apify.com/actors/development/actor-definition/output-schema" ], "maxPages": 3, "exportFormat": "markdown-and-chunks" }' | apify call agency-shift/web-content-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,agency-shift/web-content-crawler" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/GnWFKuCBdSfxRhWlu/builds/OubAhp8DSbTe3pmmk/openapi.json