# Web Scraper For Llms (`abotapi/web-scraper-for-llms`) Actor

Stealth web scraping engine built for LLMs. Converts any web page to clean markdown or HTML

- **URL**: https://apify.com/abotapi/web-scraper-for-llms.md
- **Developed by:** [Abot API](https://apify.com/abotapi) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 50 total users, 1 monthly users, 96.7% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Scraper for LLMs: Clean Markdown and HTML From Any URL

Web Scraper for LLMs turns any web page into clean markdown or HTML that is ready for a language model, a RAG pipeline, or a search index. Give it a list of URLs to scrape them directly, or turn on crawl mode and give it a starting page to follow its links and pull in everything underneath it. Every page comes back as one dataset record with its content, title and basic page details (language, favicon and canonical URL), ready to download as JSON, CSV or Excel, or to read straight through the API.

### Why This Scraper?

- **Two ways to collect pages.** Scrape a fixed list of URLs, or turn on crawl mode and let the actor follow links from a seed page to discover and read the rest of a site.
- **Markdown and HTML output.** Choose one format or both, per run, so the content slots straight into a prompt, a vector store, or a content pipeline.
- **Automatic content cleaning.** Navigation, headers, footers, sidebars, ads and tracking elements are stripped by default, so what you get is the page's actual content, not the surrounding layout.
- **Handles slow and script heavy pages.** A page is read with a quick request first, and the actor automatically falls back to full page rendering when a site needs JavaScript to show its content or blocks a simple request.
- **Page details with every record.** Title, language, favicon and canonical URL come back alongside the content, no extra parsing required.
- **Fine grained control when you need it.** CSS selectors to keep or drop specific elements, regex patterns to include or exclude URLs during a crawl, and a selector to wait for before reading a page.
- **Respects robots.txt automatically.** Disallowed URLs are skipped or refused rather than fetched, and a site's own crawl delay is honored during a crawl.

### Use Cases

- **RAG pipelines and AI knowledge bases:** turn a list of pages or a whole site into clean markdown chunks ready for embedding.
- **Documentation and content migration:** crawl a docs site or blog and export every page as markdown in one run.
- **Research and monitoring:** pull articles, product pages or listings on a recurring schedule and compare the results over time.
- **AI agent tooling:** give an agent a single action that reads any URL and returns clean text instead of raw HTML.

### Data You Get

> Sample shape: values are illustrative placeholders, not from a live record.

| Field | Example |
| --- | --- |
| `url` | `"https://example.com/blog/how-rag-pipelines-work"` |
| `title` | `"How RAG Pipelines Work"` |
| `description` | `"A short guide to building RAG pipelines."` (the page meta description, `null` when the page has none) |
| `text` | the plain readable text of the page, without markdown syntax |
| `markdown` | the page content as clean markdown (when `markdown` is a requested format) |
| `html` | the page content as cleaned HTML (when `html` is a requested format) |
| `success` | `true` |
| `error` | `null` (pages that could not be read are not written here; see the run summary) |
| `duration` | `4210` (milliseconds spent reading this page) |
| `scrapedAt` | `"2026-02-24T03:36:28.990Z"` |
| `metadata.title` | `"How RAG Pipelines Work"` |
| `metadata.language` | `"en"` |
| `metadata.favicon` | `"https://example.com/favicon.ico"` |
| `metadata.canonical` | `"https://example.com/blog/how-rag-pipelines-work"` |
| `metadata.description` | the meta description, `null` when the page has none |
| `metadata.author`, `metadata.keywords` | the author and keywords meta tags, `null` when absent |
| `metadata.languageCode` | `"en"` |
| `metadata.canonicalUrl` | `"https://example.com/blog/how-rag-pipelines-work"` |
| `metadata.openGraph`, `metadata.twitter` | the Open Graph and Twitter card tags, `null` when the page has none |
| `metadata.jsonLd` | the structured data (JSON-LD) blocks found on the page, `[]` when none |
| `metadata.headers` | the HTTP response headers of the page |
| `crawl.loadedUrl`, `crawl.loadedTime` | the final URL after redirects and when it was loaded |
| `crawl.referrerUrl`, `crawl.depth` | the page that linked here and the link hops from the seed (`0` for listed URLs) |
| `crawl.httpStatusCode`, `crawl.contentType`, `crawl.contentLengthBytes` | status code, content type and size of the response |

### How to Use

1. Add one or more URLs. Use a single seed URL if you plan to crawl from it. If you leave the URLs field out entirely, the actor scrapes four example pages, one of each shape: a bare homepage, a homepage with a query string, a static documentation page and a large wiki article. Each page is a normal result row.
2. Turn on Crawl Mode to follow links from each URL and scrape the pages it discovers, and set Crawl Depth and Max Pages to Discover to control how far it goes.
3. Choose output formats and content cleaning options, then adjust Concurrency, Max Retries and Timeout to fit the run.
4. Start the run, then download the dataset as JSON, CSV or Excel, or read it through the API.

**Scrape a short list of URLs:**

```json
{
  "urls": ["https://example.com", "https://example.com/about"]
}
```

**Scrape one page with both formats and custom content cleaning:**

```json
{
  "urls": ["https://example.com/blog/post-1"],
  "formats": ["markdown", "html"],
  "onlyMainContent": true,
  "removeAds": true,
  "excludeTags": [".newsletter-signup", ".comments"]
}
```

**Crawl a documentation site:**

```json
{
  "urls": ["https://docs.example.com"],
  "crawl": true,
  "crawlDepth": 2,
  "crawlMaxPages": 50,
  "timeoutMs": 120000,
  "includePatterns": ["/docs/"]
}
```

**Scrape several pages with a proxy and tuned concurrency:**

```json
{
  "urls": ["https://example.com/product/1", "https://example.com/product/2"],
  "concurrency": 5,
  "timeoutMs": 45000,
  "maxRetries": 3,
  "proxyConfiguration": { "useApifyProxy": true }
}
```

#### Run it from your code

Python:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("abotapi/web-scraper-for-llms").call(run_input={"urls": ["https://example.com"]})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["success"])
    print(page["markdown"])
```

JavaScript:

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('abotapi/web-scraper-for-llms').call({ urls: ['https://example.com'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

Or connect it to Make, Zapier, n8n, Google Sheets or webhooks from the **Integrations** tab.

#### URL Filters and Discovery

All settings below are optional and off or neutral by default, so existing inputs behave as before.

- **Include/Exclude URL globs:** glob versions of the regex patterns, for example `https://example.com/blog/**`. They apply to crawl-mode links and to sitemap or llms.txt URLs.
- **Use sitemaps / Use llms.txt:** also scrape the pages listed in the site's sitemap.xml or llms.txt. Those two files are fetched directly rather than through the proxy.
- **Respect robots.txt:** on by default; turn it off only when you are allowed to.
- **Max results:** stop after a total number of saved pages. Failed URLs do not count toward it. In Crawl Mode each seed may fetch up to Max Pages to Discover pages and stops saving at Max Results, so the cap counts saved rows only.
- **Only pages modified since:** skip pages whose Last-Modified header or sitemap date is older than the given date. Pages without that information are kept. In Crawl Mode, pages skipped by this filter or by incremental mode still count toward the per-seed Max Pages to Discover limit, but not toward Max Results.
- **Fetch engine:** auto (HTTP first, browser fallback), HTTP only, or browser only. Crawl-mode link discovery always uses the browser.
- **Custom HTTP headers / Initial cookies:** sent with each page request; they are not applied during crawl-mode link discovery.
- **Keep URL fragments:** keep `#fragment` on discovered links.

Failed URLs are not written to the dataset and are not billed. They are listed with their error message under `failedUrls` in the run summary (the `OUTPUT` record of the key-value store), together with per-URL success and failure counts.

### Send results into your apps (MCP connectors)

Optionally pipe a condensed summary of each scraped page into the apps you already use, via Model Context Protocol (MCP) connectors. This is an extra delivery step after the scrape: the Apify dataset is never changed, and failed pages are not exported.

1. Authorize a connector once under **Apify > Settings > Integrations** (Notion, Linear, Airtable, or Apify).
2. Select it in the **Pipe results into your apps** field (`mcpConnectors`). If the picker is empty, you have not authorized a connector yet.
3. For **Notion**, also set `notionParentPageUrl` to the page where items should be created.
4. `maxNotifyListings` caps how many pages are written to each connector per run (default 50). It does not affect the dataset.

The connection is mediated by Apify's MCP proxy, so this actor never sees your third-party credentials. Leave the field empty to skip.

### Input Parameters

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `urls` | array | sample | URLs to scrape. When Crawl Mode is on, each URL is used as a seed to discover linked pages. Required. |
| `crawl` | boolean | `false` | Follow links from each URL and scrape the pages it discovers, instead of only the URLs you listed. |
| `crawlDepth` | integer | `1` | How many link hops from the seed URL to follow. Ignored when Crawl Mode is off. |
| `crawlMaxPages` | integer | `20` | Maximum pages to discover per seed URL. Applies separately to each URL in the list. Ignored when Crawl Mode is off. |
| `formats` | array | `["markdown"]` | Content formats to return: `markdown`, `html`, or both. |
| `concurrency` | integer | `3` | Number of URLs processed in parallel. |
| `maxRetries` | integer | `2` | Retry attempts for a URL that fails to load. Ignored when Crawl Mode is on. |
| `timeoutMs` | integer | `30000` | Timeout per URL, in milliseconds. In Crawl Mode it instead caps the total link discovery time for each seed URL, so raise it for larger crawls. |
| `onlyMainContent` | boolean | `true` | Keep only the main content and strip navigation, headers, footers and sidebars. Ignored when Crawl Mode is on. |
| `removeAds` | boolean | `true` | Remove ad and tracking elements from the page. |
| `removeBase64Images` | boolean | `true` | Strip base64 encoded images to keep output size down. |
| `includeTags` | array | (none) | CSS selectors for elements to keep, everything else is dropped. Ignored when Crawl Mode is on. |
| `excludeTags` | array | (none) | CSS selectors for elements to remove from the output. Ignored when Crawl Mode is on. |
| `includePatterns` | array | (none) | Regex patterns for URL filtering. Meaningful only in Crawl Mode, where it controls which discovered links are followed. Has no effect on URLs you list directly. |
| `excludePatterns` | array | (none) | Regex patterns for URL filtering. Meaningful only in Crawl Mode, where it controls which discovered links are skipped. Has no effect on URLs you list directly. |
| `waitForSelector` | string | (none) | CSS selector to wait for before reading a page, useful for content that loads after the initial page render. Applies when the page is fully rendered. Ignored when Crawl Mode is on. |
| `incrementalMode` | boolean | `false` | Only write new or changed pages. Each page's content fingerprint is compared with the previous run; unchanged pages are skipped, not written to the dataset and not billed as results. Failed pages are always listed in the run summary. |
| `ignoreFieldsForChanges` | array | `["duration","scrapedAt","crawl.loadedTime","metadata.headers"]` | Incremental mode only. Output fields left out of the change fingerprint (they stay in the output). Dot paths reach nested keys. |
| `mcpConnectors` | array | empty | MCP connectors that receive a condensed summary of each page. |
| `notionParentPageUrl` | string | empty | Notion parent page (Notion connector only). |
| `maxNotifyListings` | integer | `50` | Max pages exported to each connector per run. |
| `crawlerType` | string | `auto` | Fetch engine: `auto` (HTTP first, browser fallback), `http` or `browser`. |
| `maxResults` | integer | 1000 | Stop after this many saved pages (0 means 1,000,000, the maximum). Failed URLs do not count. |
| `customHttpHeaders` | object | (none) | Extra HTTP headers sent with each page request. |
| `initialCookies` | array | (none) | Cookies sent with each page request. |
| `includeUrlGlobs` | array | (none) | Glob patterns a URL must match (crawl links, sitemap and llms.txt URLs). |
| `excludeUrlGlobs` | array | (none) | Glob patterns that exclude a URL. |
| `useSitemaps` | boolean | `false` | Also scrape pages listed in the site's sitemap.xml. |
| `useLlmsTxt` | boolean | `false` | Also scrape pages listed in the site's llms.txt. |
| `respectRobotsTxtFile` | boolean | `true` | Skip URLs disallowed by robots.txt. |
| `keepUrlFragments` | boolean | `false` | Keep `#fragment` on discovered links. |
| `onlyCrawlPagesModifiedSince` | string | (none) | Skip pages older than this date; pages without date information are kept. |
| `proxyConfiguration` | object | (none) | Apify proxy settings. Pages are fetched with a direct connection unless this is turned on. |

### Output Example

> Sample shape: values are illustrative placeholders, not from a live record.

```json
{
  "url": "https://example.com/blog/how-rag-pipelines-work",
  "title": "How RAG Pipelines Work",
  "description": null,
  "markdown": "# How RAG Pipelines Work\n\nRetrieval augmented generation combines a search step with a language model, so answers can draw on content the model was never trained on.",
  "html": "<article><h1>How RAG Pipelines Work</h1><p>Retrieval augmented generation combines a search step with a language model.</p></article>",
  "metadata": {
    "title": "How RAG Pipelines Work",
    "language": "en",
    "favicon": "https://example.com/favicon.ico",
    "canonical": "https://example.com/blog/how-rag-pipelines-work",
    "description": null,
    "openGraph": null,
    "twitter": null
  },
  "duration": 4210,
  "scrapedAt": "2026-02-24T03:36:28.990Z",
  "success": true,
  "error": null
}
```

### Plan Requirement

This actor runs on standard Apify compute. Pages are fetched with a direct connection unless you turn on Proxy Configuration, which uses Apify Proxy and so needs a plan that includes it. This is a general purpose reader with no single target site, and the test pages (simple public sites) loaded without a proxy. Datacenter and residential proxy lanes were not measured, so the proxy group to use is your choice: if a site blocks direct requests or leans heavily on JavaScript, try a proxy and compare results. Memory scales automatically with your URL count, crawl settings and concurrency.

### FAQ

#### How much does it cost?

Billing is pay per event: one **actor start** event per run, plus one **dataset item** event for every row written to the default dataset. The **Pricing** tab on the actor page shows the current rate for each event. URLs that could not be read, in either mode, are not written to the dataset and are not billed; they are listed under `failedUrls` in the run summary. Pages skipped as unchanged in incremental mode and pages older than the modified-since date also write no row and are not billed. Platform usage such as proxy traffic is not part of these events. Lower Max Results, Max Pages to Discover, or Crawl Depth to cap the number of rows in any run.

#### Is it legal to scrape any URL with this?

This actor is a general purpose reader, it does not choose or endorse the pages you point it at. It respects robots.txt rules and skips or refuses URLs a site has blocked. You are responsible for the URLs you provide: follow each site's terms of use and the laws that apply to you, and get legal advice before using scraped content commercially or on pages that contain personal data.

#### Can I schedule it to run automatically, or only fetch new content?

You can schedule this actor from the **Schedules** tab like any other Apify actor. By default every run scrapes the URLs you give it from scratch. Turn on **Only emit new or changed pages** (`incrementalMode`) for scheduled runs: the actor keeps a content fingerprint per page in a named key-value store and skips pages whose content is unchanged, so they are neither written to the dataset nor billed as results. The fields in `ignoreFieldsForChanges` (default `duration`, `scrapedAt`, `crawl.loadedTime` and `metadata.headers`) are not compared. The fingerprint is kept per output format, so changing formats re-emits pages once. Crawl Mode follows links breadth first, so there is no page number to resume from: each run walks the seeds again, and incremental mode only filters what is written.

#### Why did a crawl return far more pages than Max Pages to Discover?

Max Pages to Discover is a limit per seed URL, not a total for the run. If you list five seed URLs with Crawl Mode on, the actor can return up to five times that number, one batch of discovered pages per seed. Reduce the number of seed URLs, or lower Max Pages to Discover, to control the total.

#### Why did my run fail instead of returning results?

With Crawl Mode off, the run ends as failed when the input has no URLs or when no URL at all could be loaded (every URL is listed under `failedUrls` in the run summary). If a specific page cannot be read, blocked, timed out, taken down, or refused by robots.txt, it is listed under `failedUrls` with an error message, no dataset row is written for it, and the run keeps going. In Crawl Mode, pages that cannot be opened or are disallowed by robots.txt during link discovery are skipped and not listed. A crawl whose pages take more than about five minutes to read can stop the run, but rows already written are kept. Lower Max Pages to Discover or split large crawls into several runs.

#### Can I use this with AI agents or MCP?

Yes. It is built to hand language models and RAG pipelines clean markdown instead of raw HTML, and like any Apify actor it can be called from an AI agent or an MCP client through Apify's integrations.

### 🔗 Want more AI data?

Pair this actor with these related scrapers from the same team:

<table>
<tr><td>🤖 <a href="https://apify.com/abotapi/ai-fetch-python"><b>AI Agent Web Fetcher</b></a><br>An advanced web fetcher that can fetch almost all websites and convert them to...</td><td>🏨 <a href="https://apify.com/abotapi/doc-to-markdown-mcp"><b>Doc To Markdown MCP Server</b></a><br>An MCP server that converts documents to clean Markdown. Convert PDFs, Word docs, Excel...</td></tr>
<tr><td>🤖 <a href="https://apify.com/abotapi/ai-web-search-tool"><b>AI Search Tool</b></a><br>Give your AI agents real-world knowledge. This Actor provides high-quality web, news...</td><td>🤖 <a href="https://apify.com/abotapi/doc-to-markdown"><b>Doc To Markdown</b></a><br>Convert documents (PDF, Word, PowerPoint, Excel, HTML, images) to clean Markdown...</td></tr>
<tr><td>🧩 <a href="https://apify.com/abotapi/yandex-serp-scraper"><b>Yandex Search Scraper</b></a><br>Scrape Yandex web, image and video search results with URLs, snippets and organic...</td><td>⚽ <a href="https://apify.com/abotapi/hotstar-com-scraper"><b>JioHotstar Scraper</b></a><br>Scrape the JioHotstar catalog by content type or URL. Extract shows, movies, episodes...</td></tr>
</table>

👉 [Browse all abotapi scrapers](https://apify.com/abotapi)

### 💬 Support

- 🐞 **Found a bug or a missing field?** Open a ticket on the [Issues tab](https://apify.com/abotapi/web-scraper-for-llms/issues/open). We usually reply within hours.
- 🛠️ **Need another site, extra fields or a private build?** Email <contact@abotapi.com>.
- ⭐ **Enjoying it?** A quick review on the actor page helps other users find it.

# Actor input Schema

## `urls` (type: `array`):

URLs to scrape. When crawl is enabled, each URL is used as a seed to discover linked pages.

## `crawl` (type: `boolean`):

When enabled, the scraper will follow links from each URL to discover and scrape additional pages. Use Crawl Depth and Max Pages below to control how far it goes.

## `crawlDepth` (type: `integer`):

How many link hops from the starting URL. Ignored when crawl is off.

## `crawlMaxPages` (type: `integer`):

Maximum pages to discover per seed URL. Ignored when crawl is off. When left empty, a hard cap of 20 pages per seed applies; the value cannot exceed 1000.

## `formats` (type: `array`):

Which content formats to extract.

## `concurrency` (type: `integer`):

Number of URLs to process in parallel.

## `maxRetries` (type: `integer`):

Maximum retry attempts for failed URLs. Ignored when crawl is on.

## `timeoutMs` (type: `integer`):

Request timeout per URL in milliseconds.

## `waitForSelector` (type: `string`):

CSS selector to wait for before considering page loaded. Ignored when crawl is on.

## `crawlerType` (type: `string`):

auto tries a fast HTTP fetch first and falls back to a browser. http never starts a browser (fastest, no JavaScript). browser always renders the page. In crawl mode, link discovery always uses the browser.

## `maxResults` (type: `integer`):

Stop after this many pages have been saved in total (0 = the maximum of 1,000,000). Failed URLs do not count.

## `customHttpHeaders` (type: `object`):

Headers sent with each page request, as a JSON object. Not applied during crawl-mode link discovery.

## `initialCookies` (type: `array`):

Cookies sent with each page request, as a JSON array of {"name": ..., "value": ...}.

## `onlyMainContent` (type: `boolean`):

Extract only main content, removing nav/header/footer/sidebar. Ignored when crawl is on.

## `removeAds` (type: `boolean`):

Remove ads and tracking elements.

## `removeBase64Images` (type: `boolean`):

Remove base64-encoded images to reduce output size.

## `includeTags` (type: `array`):

CSS selectors for elements to include. If set, only matching elements are kept. Ignored when crawl is on.

## `excludeTags` (type: `array`):

CSS selectors for elements to exclude from output. Ignored when crawl is on.

## `includePatterns` (type: `array`):

Regex patterns for URL filtering. In crawl mode, controls which discovered links to follow.

## `excludePatterns` (type: `array`):

Regex patterns for URL filtering. In crawl mode, controls which discovered links to skip.

## `includeUrlGlobs` (type: `array`):

Only follow or scrape URLs matching one of these globs (for example https://example.com/blog/\*\*). Combined with the regex patterns above.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching one of these globs.

## `useSitemaps` (type: `boolean`):

Also scrape pages listed in the site's sitemap.xml (and Sitemap lines of robots.txt). Sitemap files are fetched directly, not through the proxy.

## `useLlmsTxt` (type: `boolean`):

Also scrape the pages linked from the site's llms.txt file. The file is fetched directly, not through the proxy.

## `respectRobotsTxtFile` (type: `boolean`):

On by default. Turn off only for sites whose robots.txt you are allowed to ignore.

## `keepUrlFragments` (type: `boolean`):

Crawl mode: keep the #fragment of discovered links instead of removing it.

## `onlyCrawlPagesModifiedSince` (type: `string`):

ISO date (2025-01-31) or relative text (7 days). Pages whose Last-Modified header or sitemap lastmod is older are skipped. Pages without that information are kept.

## `proxyConfiguration` (type: `object`):

Apify proxy configuration.

## `incrementalMode` (type: `boolean`):

Off by default. When on, each page's content is fingerprinted (title, description, markdown, html, metadata) and compared with the previous run kept in a named key-value store. Pages whose content did not change are not written to the dataset and are not billed as results. Failed pages are always reported. Fields listed in the ignore setting below (by default duration and scrapedAt) never count as a change.

## `ignoreFieldsForChanges` (type: `array`):

Output fields left out of the change fingerprint. They stay in the output. Use dot paths for nested keys, for example metadata.website.language. Defaults to duration, scrapedAt, crawl.loadedTime and metadata.headers when empty.

## `mcpConnectors` (type: `array`):

Optionally send a condensed summary of each result into the apps you already use, via Model Context Protocol (MCP) connectors. The full record stays in the dataset. Leave empty to skip. Supported: Notion, Linear, Airtable, Apify.

## `notionParentPageUrl` (type: `string`):

URL (or id) of the Notion page under which item pages are created. Required to enable the Notion export; ignored by other connectors.

## `maxNotifyListings` (type: `integer`):

Cap on items written to each connector per run. Does not affect the dataset.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "http://example.com/?a=1&b=2",
    "https://www.iana.org/domains/reserved",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "crawl": false,
  "crawlDepth": 1,
  "formats": [
    "markdown"
  ],
  "concurrency": 3,
  "maxRetries": 2,
  "timeoutMs": 30000,
  "crawlerType": "auto",
  "maxResults": 1000,
  "onlyMainContent": true,
  "removeAds": true,
  "removeBase64Images": true,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": true,
  "keepUrlFragments": false,
  "incrementalMode": false,
  "ignoreFieldsForChanges": [
    "duration",
    "scrapedAt",
    "crawl.loadedTime",
    "metadata.headers"
  ],
  "maxNotifyListings": 50
}
```

# Actor output Schema

## `scrapedPages` (type: `string`):

Each scraped page as a dataset row with URL, content, metadata, and status.

## `details` (type: `string`):

HTTP status, depth, content type and page metadata for each page.

## `runSummary` (type: `string`):

Per-URL success and failure counts, plus the list of URLs that could not be read (failedUrls). Failed URLs are not written to the dataset and are not billed.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "http://example.com/?a=1&b=2",
        "https://www.iana.org/domains/reserved",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ],
    "ignoreFieldsForChanges": [
        "duration",
        "scrapedAt",
        "crawl.loadedTime",
        "metadata.headers"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("abotapi/web-scraper-for-llms").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://example.com",
        "http://example.com/?a=1&b=2",
        "https://www.iana.org/domains/reserved",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ],
    "ignoreFieldsForChanges": [
        "duration",
        "scrapedAt",
        "crawl.loadedTime",
        "metadata.headers",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("abotapi/web-scraper-for-llms").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "http://example.com/?a=1&b=2",
    "https://www.iana.org/domains/reserved",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "ignoreFieldsForChanges": [
    "duration",
    "scrapedAt",
    "crawl.loadedTime",
    "metadata.headers"
  ]
}' |
apify call abotapi/web-scraper-for-llms --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,abotapi/web-scraper-for-llms"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fkaqHAofsFCxthymB/builds/oUhd7MGtzQME9ANm0/openapi.json