# AI Web Scraper - Powered by Crawl4AI (`raizen/ai-web-scraper`) Actor A blazing-fast AI web scraper powered by Crawl4AI. Perfect for LLMs, AI agents, AI automation, model training, sentiment analysis, and content generation. Supports deep crawling, multiple extraction strategies and flexible output (Markdown/JSON). Seamlessly integrates with Make.com, n8n, and Zapier. - **URL**: https://apify.com/raizen/ai-web-scraper.md - **Developed by:** [Raizen Technology](https://apify.com/raizen) (community) - **Categories:** AI, Agents, Automation - **Features:** Full permissions - **Stats:** 357 total users, 2 monthly users, 100.0% runs succeeded, 11 bookmarks - **User rating**: 1.00 out of 5 stars ## Pricing $8.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## AI Web Scraper Do you need reliable data for your AI agents, LLM pipelines, or training workflows? The **AI Web Scraper** Actor is your key to fast, flexible, and AI‑friendly web extraction on Apify. Under the hood, it relies on the open‑source [Crawl4AI](https://github.com/unclecode/crawl4ai) engine to handle anything from simple single‑page scrapes to deep multi‑link traversals (BFS/DFS/BestFirst). Whether you want clean markdown, JSON extraction, or LLM summarization, just specify your desired strategy via the Actor’s input UI, and you’re set. This Actor integrates well with Make.com, n8n, and Zapier for AI automation Below is an overview of each setting you’ll see in the Apify interface and how it affects your crawls. *** ### Quick How-To 1. **Start with URLs**\ At a minimum, provide `startUrls` in the UI (or JSON input)—the pages you want to scrape. 2. **Pick a Crawler & Extraction Style**\ Choose between various crawl strategies (e.g., BFS or DFS) and extraction methods (simple markdown, LLM-based, JSON CSS, etc.). You can also enable content filtering or deeper link exploration. 3. **Review the Output**\ Once the Actor finishes, your results will appear in the Apify Dataset—structured JSON, markdown, or whichever format you chose. *** ### Input Fields Explained These fields appear in the Actor’s input UI. Customize them to match your use case. #### 1. **startUrls** (Required) List of pages to scrape. For each entry, just provide `"url"`. **Example**: ```json { "startUrls": [ { "url": "https://example.com" } ] } ``` #### 2. **browserConfig** (Optional) Configure Playwright’s browser behavior—headless mode, custom user agent, viewport size, etc. - `browser_type`: “chromium”, “firefox”, or “webkit” - `headless`: Boolean to run in headless mode - `verbose_logging`: Extra debug logs - `ignore_https_errors`: Accept invalid certs - `user_agent`: E.g. “random” or a custom string - `proxy`: Proxy server URL - `viewport_width` / `viewport_height`: Window size - `accept_downloads`: Whether downloads are allowed - `extra_headers`: Additional request headers #### 3. **crawlerConfig** (Optional) Core crawling settings—time limits, caching, JavaScript hooks, or multi‑page concurrency. - `cache_mode`: “BYPASS” (no cache), “ENABLED”, etc. - `page_timeout`: Milliseconds to wait for page loads - `simulate_user`: Stealth by mimicking user actions - `remove_overlay_elements`: Attempt to remove popups - `delay_before_return_html`: Extra wait before final extraction - `wait_for`: Wait time or wait condition - `screenshot` / `pdf`: Capture screenshot or PDF - `enable_rate_limiting`: Rate limit large URL lists - `memory_threshold_percent`: Pause if memory is too high - `word_count_threshold`: Discard short text blocks - `css_selector`, `excluded_tags`, `excluded_selector`: Further refine or skip sections of the DOM - `only_text`: Keep plain text only - `prettify`: Attempt to clean up HTML - `keep_data_attributes`: Keep or drop data-\* attributes - `remove_forms`: Strip `
` elements - `bypass_cache` / `disable_cache` / `no_cache_read` / `no_cache_write`: Fine‑grained caching controls - `wait_until`: E.g. “domcontentloaded” or “networkidle” - `wait_for_images`: Wait for images to fully load - `check_robots_txt`: Respect robots.txt? - `mean_delay`, `max_range`: Introduce a random delay range - `js_code`: Custom JS to run on each page - `js_only`: Reuse the same page context without re-navigation - `ignore_body_visibility`: Include hidden elements - `scan_full_page`: Scroll from top to bottom for lazy loading - `scroll_delay`: Delay between scroll steps - `process_iframes`: Also parse iframes - `override_navigator`: Additional stealth tweak - `magic`: Enable multiple advanced anti-bot tricks - `adjust_viewport_to_content`: Resize viewport to fit content - `screenshot_wait_for`: Wait time before taking a screenshot - `screenshot_height_threshold`: Max doc height to screenshot - `image_description_min_word_threshold`: Filter out images with minimal alt text - `image_score_threshold`: Remove lower‑score images - `exclude_external_images`: No external images - `exclude_social_media_domains`, `exclude_domains`: Avoid these domains entirely - `exclude_external_links`, `exclude_social_media_links`: Strip external or social media links - `verbose`: Extra logs - `log_console`: Show browser console logs? - `stream`: Stream results as they come in, or wait until done #### 4. **deepCrawlConfig** (Optional) When you select BFS, DFS, or BestFirst crawling, this config guides link exploration. - `max_pages`: Stop after crawling this many pages - `max_depth`: Depth of link‑following - `include_external`: Follow off‑domain links? - `score_threshold`: Filter out low‑score links (BestFirst) - `filter_chain`: Extra link filter rules - `url_scorer`: If you want a custom approach to scoring discovered URLs #### 5. **markdownConfig** (Optional) For HTML→Markdown conversions. - `ignore_links`: Skip anchor links - `ignore_images`: Omit markdown images - `escape_html`: Turn `
` into `<div>` - `skip_internal_links`: Remove same‑page anchors - `include_sup_sub`: Preserve `/` text - `citations`: Put footnotes at bottom of file - `body_width`: Wrap lines at N chars - `fit_markdown`: Use advanced “fit” mode if also using a filter #### 6. **contentFilterConfig** (Optional) Prune out nav bars, sidebars, or extra text using “pruning,” “bm25,” or a second LLM filter. - `type`: e.g. “pruning”, “bm25” - `threshold`: Score cutoff - `min_word_threshold`: Minimum words to keep - `bm25_threshold`: BM25 filter param - `apply_llm_filter`: If `true`, do a second pass with an LLM - `semantic_filter`: Keep only text about a certain topic - `word_count_threshold`: Another word threshold - `sim_threshold`, `max_dist`, `top_k`, `linkage_method`: For advanced clustering #### 7. **userAgentConfig** (Optional) Rotate or fix your user agent. - `user_agent_mode`: “random” or “fixed” - `device_type`: “desktop” or “mobile” - `browser_type`: e.g. “chrome” - `num_browsers`: If rotating among multiple agents #### 8. **llmConfig** (Optional) For LLM-based extraction or filtering. - `provider`: e.g. “openai/gpt-4”, “groq/deepseek-r1-distill-llama-70b” - `api_token`: Model’s API key - `instruction`: Prompt the LLM about how to parse or summarize - `base_url`: For custom endpoints - `chunk_token_threshold`: Big pages → chunk them - `apply_chunking`: Boolean for chunking - `input_format`: “markdown” or “html” - `temperature`, `max_tokens`: Standard LLM config #### 9. **session_id** (Optional) Provide a session ID to reuse browser context across multiple runs (logins, multi-step flows, etc.). #### 10. **extractionStrategy** (Optional) Pick one: - `SimpleExtractionStrategy`: Simple HTML→Markdown. - `LLMExtractionStrategy`: Let an LLM parse or summarize. - `JsonCssExtractionStrategy` / `JsonXPathExtractionStrategy`: Provide a schema to produce structured JSON. #### 11. **crawlStrategy** (Optional) - `SimpleCrawlStrategy`: Just the given start URLs - `BFSDeepCrawlStrategy`: Breadth-first approach - `DFSDeepCrawlStrategy`: Depth-first approach - `BestFirstCrawlingStrategy`: Score links, pick the best first #### 12. **extractionSchema** (Optional) If using CSS/XPath extraction. - `name`: Your extraction scheme name - `baseSelector`: Parent selector for repeated elements - `fields`: Each with `name`, `selector`, `type`, and optional `attribute` **Example**: ```json "extractionSchema": { "name": "Custom Extraction", "baseSelector": "div.article", "fields": [ { "name": "title", "selector": "h1", "type": "text" }, { "name": "link", "selector": "a", "type": "attribute", "attribute": "href" } ] } ``` *** ### Usage Examples #### Minimal ```json { "startUrls": [ { "url": "https://example.com" } ] } ``` Scrapes a single page in headless mode with standard markdown output. #### JSON CSS Extraction ```json { "startUrls": [ { "url": "https://news.ycombinator.com/" } ], "extractionStrategy": "JsonCssExtractionStrategy", "extractionSchema": { "name": "HackerNews", "baseSelector": "tr.athing", "fields": [ { "name": "title", "selector": ".titleline a", "type": "text" }, { "name": "link", "selector": ".titleline a", "type": "attribute", "attribute": "href" } ] } } ``` Generates a JSON array, each object containing “title” and “link.” *** ### Pro Tips - **Deep crawling**: If you want BFS or DFS, set `crawlStrategy` to “BFSDeepCrawlStrategy” or “DFSDeepCrawlStrategy” and configure `deepCrawlConfig`. - **Content filtering**: Combine `contentFilterConfig` with `extractionStrategy` for maximum clarity and minimal noise. - **LLM-based**: Choose “LLMExtractionStrategy” plus `llmConfig` for advanced summarization or structured data. Great for building AI pipelines. Thanks for trying out the **AI Web Scraper**—enjoy harnessing rich, clean data for your Apify-based AI solutions! # Actor input Schema ## `startUrls` (type: `array`): List of webpages to scrape. ## `extractionStrategy` (type: `string`): Select how content is extracted. ## `crawlStrategy` (type: `string`): Select how pages are crawled. ## `browserConfig` (type: `object`): Browser settings as JSON object. ## `crawlerConfig` (type: `object`): Crawler settings as JSON object. ## `deepCrawlConfig` (type: `object`): Settings for deep crawling when using BFS, DFS, or Best-First Strategies. ## `markdownConfig` (type: `object`): Markdown settings as JSON object. ## `contentFilterConfig` (type: `object`): Content filter settings as JSON object. ## `userAgentConfig` (type: `object`): User agent settings for browser requests. ## `llmConfig` (type: `object`): Configure LLM usage for content extraction. ## `extractionSchema` (type: `object`): Define custom extraction rules when using JsonCssExtractionStrategy or JsonXPathExtractionStrategy. ## `session_id` (type: `string`): Use a session ID to persist browser state across multiple requests. ## Actor input object example ```json { "startUrls": [ { "url": "https://www.cnbc.com/2025/03/12/googles-deepmind-says-it-will-use-ai-models-to-power-physical-robots.html" } ], "extractionStrategy": "SimpleExtractionStrategy", "crawlStrategy": "SimpleCrawlStrategy", "browserConfig": { "browser_type": "chromium", "headless": true, "verbose_logging": false, "ignore_https_errors": true, "user_agent": "random", "proxy": "", "viewport_width": 1280, "viewport_height": 720, "accept_downloads": false, "extra_headers": {} }, "crawlerConfig": { "cache_mode": "BYPASS", "page_timeout": 20000, "simulate_user": true, "override_navigator": true, "magic": true, "remove_overlay_elements": true, "delay_before_return_html": 0.75, "wait_for": "", "screenshot": false, "pdf": false, "enable_rate_limiting": false, "memory_threshold_percent": 90, "word_count_threshold": 200, "css_selector": "", "excluded_tags": [], "excluded_selector": "", "only_text": false, "prettify": false, "keep_data_attributes": false, "remove_forms": false, "bypass_cache": false, "disable_cache": false, "no_cache_read": false, "no_cache_write": false, "wait_until": "domcontentloaded", "wait_for_images": false, "check_robots_txt": false, "mean_delay": 0.1, "max_range": 0.3, "js_code": "", "js_only": false, "ignore_body_visibility": true, "scan_full_page": false, "scroll_delay": 0.2, "process_iframes": false, "adjust_viewport_to_content": false, "screenshot_wait_for": 0, "screenshot_height_threshold": 20000, "image_description_min_word_threshold": 50, "image_score_threshold": 3, "exclude_external_images": false, "exclude_social_media_domains": [], "exclude_external_links": false, "exclude_social_media_links": false, "exclude_domains": [], "verbose": true, "log_console": false, "stream": false }, "deepCrawlConfig": { "max_pages": 100, "max_depth": 3, "include_external": false, "score_threshold": 0.5, "filter_chain": [], "keywords": [ "crawl", "example", "async", "configuration" ], "weight": 0.7 }, "markdownConfig": { "ignore_links": false, "ignore_images": false, "escape_html": true, "skip_internal_links": false, "include_sup_sub": false, "citations": false, "body_width": 80, "fit_markdown": false }, "contentFilterConfig": { "type": "pruning", "user_query": "", "threshold": 0.45, "min_word_threshold": 5, "bm25_threshold": 1.2, "apply_llm_filter": false, "semantic_filter": "", "word_count_threshold": 10, "sim_threshold": 0.3, "max_dist": 0.2, "top_k": 3, "linkage_method": "ward" }, "userAgentConfig": { "user_agent_mode": "random", "device_type": "desktop", "browser_type": "chrome", "num_browsers": 1 }, "llmConfig": { "provider": "groq/deepseek-r1-distill-llama-70b", "api_token": "", "instruction": "Summarize content in clean markdown.", "base_url": "", "chunk_token_threshold": 2048, "apply_chunking": true, "input_format": "markdown", "temperature": 0.7, "max_tokens": 4096 }, "extractionSchema": { "name": "Custom Extraction", "baseSelector": "div.article", "fields": [ { "name": "title", "selector": "h1", "type": "text" }, { "name": "link", "selector": "a", "type": "attribute", "attribute": "href" } ] }, "session_id": "" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://www.cnbc.com/2025/03/12/googles-deepmind-says-it-will-use-ai-models-to-power-physical-robots.html" } ], "browserConfig": { "browser_type": "chromium", "headless": true, "verbose_logging": false, "ignore_https_errors": true, "user_agent": "random", "proxy": "", "viewport_width": 1280, "viewport_height": 720, "accept_downloads": false, "extra_headers": {} }, "crawlerConfig": { "cache_mode": "BYPASS", "page_timeout": 20000, "simulate_user": true, "override_navigator": true, "magic": true, "remove_overlay_elements": true, "delay_before_return_html": 0.75, "wait_for": "", "screenshot": false, "pdf": false, "enable_rate_limiting": false, "memory_threshold_percent": 90, "word_count_threshold": 200, "css_selector": "", "excluded_tags": [], "excluded_selector": "", "only_text": false, "prettify": false, "keep_data_attributes": false, "remove_forms": false, "bypass_cache": false, "disable_cache": false, "no_cache_read": false, "no_cache_write": false, "wait_until": "domcontentloaded", "wait_for_images": false, "check_robots_txt": false, "mean_delay": 0.1, "max_range": 0.3, "js_code": "", "js_only": false, "ignore_body_visibility": true, "scan_full_page": false, "scroll_delay": 0.2, "process_iframes": false, "adjust_viewport_to_content": false, "screenshot_wait_for": 0, "screenshot_height_threshold": 20000, "image_description_min_word_threshold": 50, "image_score_threshold": 3, "exclude_external_images": false, "exclude_social_media_domains": [], "exclude_external_links": false, "exclude_social_media_links": false, "exclude_domains": [], "verbose": true, "log_console": false, "stream": false }, "deepCrawlConfig": { "max_pages": 100, "max_depth": 3, "include_external": false, "score_threshold": 0.5, "filter_chain": [], "keywords": [ "crawl", "example", "async", "configuration" ], "weight": 0.7 }, "markdownConfig": { "ignore_links": false, "ignore_images": false, "escape_html": true, "skip_internal_links": false, "include_sup_sub": false, "citations": false, "body_width": 80, "fit_markdown": false }, "contentFilterConfig": { "type": "pruning", "user_query": "", "threshold": 0.45, "min_word_threshold": 5, "bm25_threshold": 1.2, "apply_llm_filter": false, "semantic_filter": "", "word_count_threshold": 10, "sim_threshold": 0.3, "max_dist": 0.2, "top_k": 3, "linkage_method": "ward" }, "userAgentConfig": { "user_agent_mode": "random", "device_type": "desktop", "browser_type": "chrome", "num_browsers": 1 }, "llmConfig": { "provider": "groq/deepseek-r1-distill-llama-70b", "api_token": "", "instruction": "Summarize content in clean markdown.", "base_url": "", "chunk_token_threshold": 2048, "apply_chunking": true, "input_format": "markdown", "temperature": 0.7, "max_tokens": 4096 }, "extractionSchema": { "name": "Custom Extraction", "baseSelector": "div.article", "fields": [ { "name": "title", "selector": "h1", "type": "text" }, { "name": "link", "selector": "a", "type": "attribute", "attribute": "href" } ] } }; // Run the Actor and wait for it to finish const run = await client.actor("raizen/ai-web-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://www.cnbc.com/2025/03/12/googles-deepmind-says-it-will-use-ai-models-to-power-physical-robots.html" }], "browserConfig": { "browser_type": "chromium", "headless": True, "verbose_logging": False, "ignore_https_errors": True, "user_agent": "random", "proxy": "", "viewport_width": 1280, "viewport_height": 720, "accept_downloads": False, "extra_headers": {}, }, "crawlerConfig": { "cache_mode": "BYPASS", "page_timeout": 20000, "simulate_user": True, "override_navigator": True, "magic": True, "remove_overlay_elements": True, "delay_before_return_html": 0.75, "wait_for": "", "screenshot": False, "pdf": False, "enable_rate_limiting": False, "memory_threshold_percent": 90, "word_count_threshold": 200, "css_selector": "", "excluded_tags": [], "excluded_selector": "", "only_text": False, "prettify": False, "keep_data_attributes": False, "remove_forms": False, "bypass_cache": False, "disable_cache": False, "no_cache_read": False, "no_cache_write": False, "wait_until": "domcontentloaded", "wait_for_images": False, "check_robots_txt": False, "mean_delay": 0.1, "max_range": 0.3, "js_code": "", "js_only": False, "ignore_body_visibility": True, "scan_full_page": False, "scroll_delay": 0.2, "process_iframes": False, "adjust_viewport_to_content": False, "screenshot_wait_for": 0, "screenshot_height_threshold": 20000, "image_description_min_word_threshold": 50, "image_score_threshold": 3, "exclude_external_images": False, "exclude_social_media_domains": [], "exclude_external_links": False, "exclude_social_media_links": False, "exclude_domains": [], "verbose": True, "log_console": False, "stream": False, }, "deepCrawlConfig": { "max_pages": 100, "max_depth": 3, "include_external": False, "score_threshold": 0.5, "filter_chain": [], "keywords": [ "crawl", "example", "async", "configuration", ], "weight": 0.7, }, "markdownConfig": { "ignore_links": False, "ignore_images": False, "escape_html": True, "skip_internal_links": False, "include_sup_sub": False, "citations": False, "body_width": 80, "fit_markdown": False, }, "contentFilterConfig": { "type": "pruning", "user_query": "", "threshold": 0.45, "min_word_threshold": 5, "bm25_threshold": 1.2, "apply_llm_filter": False, "semantic_filter": "", "word_count_threshold": 10, "sim_threshold": 0.3, "max_dist": 0.2, "top_k": 3, "linkage_method": "ward", }, "userAgentConfig": { "user_agent_mode": "random", "device_type": "desktop", "browser_type": "chrome", "num_browsers": 1, }, "llmConfig": { "provider": "groq/deepseek-r1-distill-llama-70b", "api_token": "", "instruction": "Summarize content in clean markdown.", "base_url": "", "chunk_token_threshold": 2048, "apply_chunking": True, "input_format": "markdown", "temperature": 0.7, "max_tokens": 4096, }, "extractionSchema": { "name": "Custom Extraction", "baseSelector": "div.article", "fields": [ { "name": "title", "selector": "h1", "type": "text", }, { "name": "link", "selector": "a", "type": "attribute", "attribute": "href", }, ], }, } # Run the Actor and wait for it to finish run = client.actor("raizen/ai-web-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://www.cnbc.com/2025/03/12/googles-deepmind-says-it-will-use-ai-models-to-power-physical-robots.html" } ], "browserConfig": { "browser_type": "chromium", "headless": true, "verbose_logging": false, "ignore_https_errors": true, "user_agent": "random", "proxy": "", "viewport_width": 1280, "viewport_height": 720, "accept_downloads": false, "extra_headers": {} }, "crawlerConfig": { "cache_mode": "BYPASS", "page_timeout": 20000, "simulate_user": true, "override_navigator": true, "magic": true, "remove_overlay_elements": true, "delay_before_return_html": 0.75, "wait_for": "", "screenshot": false, "pdf": false, "enable_rate_limiting": false, "memory_threshold_percent": 90, "word_count_threshold": 200, "css_selector": "", "excluded_tags": [], "excluded_selector": "", "only_text": false, "prettify": false, "keep_data_attributes": false, "remove_forms": false, "bypass_cache": false, "disable_cache": false, "no_cache_read": false, "no_cache_write": false, "wait_until": "domcontentloaded", "wait_for_images": false, "check_robots_txt": false, "mean_delay": 0.1, "max_range": 0.3, "js_code": "", "js_only": false, "ignore_body_visibility": true, "scan_full_page": false, "scroll_delay": 0.2, "process_iframes": false, "adjust_viewport_to_content": false, "screenshot_wait_for": 0, "screenshot_height_threshold": 20000, "image_description_min_word_threshold": 50, "image_score_threshold": 3, "exclude_external_images": false, "exclude_social_media_domains": [], "exclude_external_links": false, "exclude_social_media_links": false, "exclude_domains": [], "verbose": true, "log_console": false, "stream": false }, "deepCrawlConfig": { "max_pages": 100, "max_depth": 3, "include_external": false, "score_threshold": 0.5, "filter_chain": [], "keywords": [ "crawl", "example", "async", "configuration" ], "weight": 0.7 }, "markdownConfig": { "ignore_links": false, "ignore_images": false, "escape_html": true, "skip_internal_links": false, "include_sup_sub": false, "citations": false, "body_width": 80, "fit_markdown": false }, "contentFilterConfig": { "type": "pruning", "user_query": "", "threshold": 0.45, "min_word_threshold": 5, "bm25_threshold": 1.2, "apply_llm_filter": false, "semantic_filter": "", "word_count_threshold": 10, "sim_threshold": 0.3, "max_dist": 0.2, "top_k": 3, "linkage_method": "ward" }, "userAgentConfig": { "user_agent_mode": "random", "device_type": "desktop", "browser_type": "chrome", "num_browsers": 1 }, "llmConfig": { "provider": "groq/deepseek-r1-distill-llama-70b", "api_token": "", "instruction": "Summarize content in clean markdown.", "base_url": "", "chunk_token_threshold": 2048, "apply_chunking": true, "input_format": "markdown", "temperature": 0.7, "max_tokens": 4096 }, "extractionSchema": { "name": "Custom Extraction", "baseSelector": "div.article", "fields": [ { "name": "title", "selector": "h1", "type": "text" }, { "name": "link", "selector": "a", "type": "attribute", "attribute": "href" } ] } }' | apify call raizen/ai-web-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,raizen/ai-web-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/zOQWQaziNeBNFWN1O/builds/yLnHaI297EvKjolLE/openapi.json