# Website Intelligence Crawler (`dami_studio/website-intelligence-crawler`) Actor

Website intelligence crawler: crawl a public website and get its pages as clean text and Markdown, plus chunks sized for embedding with token estimates, title, description, language and same-site links. Follows robots.txt. No model, no browser. $0.60 per 1,000 pages.

- **URL**: https://apify.com/dami_studio/website-intelligence-crawler.md
- **Developed by:** [Dami's Studio](https://apify.com/dami_studio) (community)
- **Categories:** Developer tools, SEO tools, AI
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.57 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Website crawler for clean text, Markdown and chunks sized for an embedding model, from a public site**: title, description, language, word count, the same-site links and the status code ride along on every row. No API key, no model call, nothing to authorise.

The one limit to understand first: there is no browser here. It reads the HTML the server sends, so a page that builds its content in JavaScript after loading comes back with very little text, and no setting changes that. For ordinary documentation, blogs, marketing sites and knowledge bases, the HTML is the content.

| | |
|---|---|
| **Input** | Public HTTP or HTTPS URLs to start from |
| **Output** | One row per page: readable text, Markdown, chunks with token estimates, title, description, language, links, status |
| **Ceiling** | 1,000 pages per starting URL, 20 levels of links |
| **Account needed** | None |
| **Price** | $0.60 per 1,000 pages ($0.0006 each) on the Free plan, a little less on paid plans. The free plan's $5 a month covers about 8,300 |

### 🔍 What Website Crawler does

From each URL you give it, it follows same-site links down to the depth you set and writes a record per page. It stays on the site it started on unless you ask it to read the websites the site links to.

You can aim it: URL patterns to follow or skip, the start folder only, subdomains, the sitemap or llms.txt as extra starting points, and one page limit for the whole run.

The text is the readable part of the page, with navigation, scripts and boilerplate stripped. The Markdown keeps the heading structure, paragraphs and lists, which is the version you want if the pages are going into a document store.

Chunks are cut to the size you ask for with a little overlap, so a sentence is not sliced in half. Each one carries its character count and an estimated token count, so you can budget an embedding run before you start it.

Give it a `query` and every chunk gets a `relevanceScore` against those words. It is plain term matching, not a model, and it is there to let you keep the 20 chunks that matter instead of all 400. The chunking and the scoring are both arithmetic, so the same input gives the same output every time.

### 📋 What data you get from each page

| What you get | Field |
|---|---|
| The page's readable text | `text` |
| The same content as Markdown, headings kept | `markdown` |
| Chunks with character count, token estimate and relevance score | `chunks` |
| Title, description, language and canonical address | `title`, `description`, `language`, `canonicalUrl` |
| Same-site links with their anchor text | `links` |
| Word count and HTTP status | `wordCount`, `status` |
| Where it came from: the starting URL, the site, the link depth and the page that linked here | `seedUrl`, `siteKey`, `depth`, `referrerUrl` |
| The address asked for, and the one that answered after redirects | `requestedUrl`, `url`, `finalUrl` |
| Open Graph, site name, sharing image, published date, keywords | `openGraph`, `siteName`, `imageUrl`, `publishedAt`, `keywords` |
| Headings with their level | `headings` |
| Links to other websites with anchor text, and link totals by place on the page | `externalLinks`, `linkStats` |
| Robots tag, noindex and where the canonical points | `indexability` |
| Structured data: its types and the JSON-LD blocks as published | `schemaTypes`, `jsonLd` |
| Images with alt text and declared size | `images`, `imageStats` |
| Paragraph and sentence counts, reading ease, top words | `contentStats` |
| Response headers and time | `responseHeaders`, `responseTimeMs` |
| A flag for pages that only fill in once scripts run | `needsJavaScript` |
| Pages in the run sharing a title or description | `duplicateTitleOf`, `duplicateDescriptionOf` |
| Optional: cleaned HTML, published e-mails and phone numbers | `html`, `emails`, `phones` |

### ▶️ How to crawl a website into text, Markdown and chunks

1. Open [Website Crawler](https://apify.com/dami_studio/website-intelligence-crawler) and click **Try for free**.
2. Put your address into **Start URLs**.
3. Set **Maximum pages** and **Maximum link depth**. Ten pages at depth 1 is a good look before you commit to a whole site.
4. If the crawl is feeding a search or a RAG index, set **Chunk size** to whatever your embedding model likes.
5. Click **Start**, then download the dataset as JSON, CSV or Excel, or read it from the API.

Leave the URLs empty and you get one sample row showing the shape, with nothing fetched and nothing charged.

### 💰 How much does it cost to crawl a website?

**$0.60 per 1,000 pages**, which is $0.0006 each on the Free plan. Paid plans pay a little less: $0.59 on Starter, $0.58 on Scale and $0.57 on Business. The free plan's $5 a month covers about 8,300 pages.

You pay per page that came back. The sample row is not charged, and neither is a page that failed, was disallowed by `robots.txt`, was too large, or turned out to be a PDF. Those still arrive as rows so you know they happened. Summary rows are free, and so are pages the two optional add-ons leave out until 9 November 2026. From that day:

| Add-on | Price |
|---|---|
| `skipCanonicalDuplicates`, `changedSince` | $0.30 per 1,000 pages they fetch and leave out on the Free plan, a little less on paid plans |

Fifty pages is three cents. A hundred-page site crawled weekly is about a quarter a month. If you set a maximum charge for a run, it stops at the last page that fits, and the run's status says so.

### 📥 What you give it

```json
{
  "startUrls": [{"url": "https://docs.example.com/"}],
  "maxPages": 50,
  "maxDepth": 2,
  "outputFormat": "both",
  "chunkSizeChars": 1200,
  "query": "pricing plans billing"
}
```

| Field | Default | What it is |
|---|---|---|
| `startUrls` | none | The pages to start from. Normalised and deduplicated. The Console box starts with `https://example.com/`. |
| `urls` | none | More starting URLs, as plain strings, if that shape suits you better. |
| `maxPages` | `10` | Records per starting URL, 1 to 1,000. Failed pages count too, unless `countOnlyGoodPages` is on. |
| `maxDepth` | `1` | Link depth, 0 to 20. 0 reads the starting pages only. |
| `maxConcurrency` | `3` | Pages fetched at once from each site, 1 to 10. Several starting URLs are also crawled side by side up to the same number, so 3 with three starting URLs can mean nine requests at once. Leave it low on small servers. |
| `useRobotsTxt` | `true` | Honour Disallow rules and crawl delay. The delay is kept between batches of pages, so use `maxConcurrency` 1 for a site that asks for one. Allow exceptions are not read, so a site that relies on them may be crawled less than it permits. |
| `query` | none | Words to score chunks against. No model, no key. |
| `chunkSizeChars` | `1200` | Target characters per chunk, 300 to 5,000. |
| `chunkOverlapChars` | `120` | Characters repeated between neighbouring chunks, up to 800. Zero behaves as 120. |
| `outputFormat` | `both` | `json` keeps the text, `markdown` keeps the Markdown, `both` keeps each. |
| `requestTimeoutSecs` | `15` | Per request, 3 to 60. |
| `maxResponseSizeKb` | `2048` | Biggest page body accepted, 128 to 8,192 KB. |
| `fallbackToProxy` | `false` | Retry a page that could not be read. |
| `proxyConfiguration` | none | Your own servers, if you have them. |
| `maxPagesPerRun` | none | One page limit for the whole run, across the starting URLs. |
| `includeUrlGlobs` | none | Follow only links matching these patterns, such as `/docs/**`. `*` stays inside a folder, `**` crosses folders. Up to 20. Starting URLs are always read. |
| `excludeUrlGlobs` | none | Never follow links matching these patterns, such as `**?page=*`. |
| `stayUnderStartPath` | `false` | A crawl started at `/docs/` stays in `/docs/`. |
| `includeSubdomains` | `false` | `docs.example.com` counts as part of `example.com`. |
| `followNavigationLinks` | `false` | Also follow links in menus, headers and footers. |
| `skipFileLinks` | `false` | Don't request links ending in a file type: PDFs, images, archives, feeds. |
| `countOnlyGoodPages` | `false` | Failed and left-out pages don't use up `maxPages`; tries stop at twice the limit. |
| `useSitemap` | `false` | Also start from the sitemap, indexes included. A starting URL ending in `.xml` is read as a sitemap. |
| `useLlmsTxt` | `false` | Also start from the pages in the site's `llms.txt`, Markdown pages read as Markdown. |
| `followExternalLinks` | `false` | Also read pages on websites the site links to, one step away, their `robots.txt` honoured, sign-in and checkout addresses skipped. |
| `maxPagesPerOtherSite` | `3` | Pages per other website, 1 to 20. |
| `requestLanguage` | US English | The language list sent with each request, such as `de-DE,de;q=0.9`. |
| `markdownStyle` | `basic` | `rich` keeps links, bold, tables and code blocks too. Chunks are always cut from the plain text. |
| `contentExtractor` | `default` | `default` reads the main or article element, `article` picks the block with the most paragraph text, `wholePage` reads the body. |
| `keepSelector` | none | Keep only what matches this CSS selector. No match keeps the normal text and sets `keepSelectorMatched` to false. |
| `removeSelector` | none | Remove what matches this CSS selector before the text is read. |
| `removeCookieBanners` | `false` | Remove cookie and consent dialogs from the text. |
| `dropRepeatedLines` | `false` | Drop text blocks an earlier page of the same site already had; the text is rebuilt block by block. |
| `includeHtml` | `false` | Add the cleaned HTML of the content as `html`. Rows get several times bigger. |
| `includeContacts` | `false` | Add the e-mails and tap-to-call numbers the site publishes, as `emails` and `phones`. Not read on other websites. |
| `saveSkippedUrls` | `false` | Save the links your rules left out, with the reason, in the key-value record `SKIPPED_URLS`. |
| `skipCanonicalDuplicates` | `false` | Optional add-on: leave out a page whose canonical address was already crawled in the run. |
| `changedSince` | none | Optional add-on for repeat crawls: `2026-10-01` or `7 days`. Pages the server reports unchanged are left out and sitemap entries dated earlier are not requested. Pair it with `useSitemap`, since an unchanged page sent without a copy is not read for links. |

### 📤 What you get back

A real row from a run on 27 September 2026, the first of five pages read from python.org. The text, the Markdown and the chunk texts are cut short, and the links list is cut to two of fifteen. No `query` was set, so `relevanceScore` is `null`.

```json
{
  "ok": true,
  "requestedUrl": "https://python.org/",
  "url": "https://python.org/",
  "seedUrl": "https://python.org/",
  "siteKey": "python.org",
  "depth": 0,
  "status": 200,
  "title": "Welcome to Python.org",
  "description": "The official home of the Python Programming Language",
  "language": "en",
  "canonicalUrl": "https://python.org/",
  "text": "Get Started\nWhether you're new to programming or an experienced developer, it's easy to learn and use Python. ...",
  "markdown": "## Get Started\n\nWhether you're new to programming or an experienced developer, it's easy to learn and use Python.\n\n...",
  "links": [
    { "url": "https://python.org/about/gettingstarted/", "text": "Start with our Beginner’s Guide" },
    { "url": "https://python.org/downloads/release/python-3147/", "text": "Python 3.14.7" }
  ],
  "wordCount": 295,
  "contentType": "text/html",
  "chunks": [
    { "chunkIndex": 0, "text": "Get Started\nWhether you're new to programming ...", "charCount": 1200, "estimatedTokens": 300, "relevanceScore": null },
    { "chunkIndex": 1, "text": "26\n\nSuccess Stories\nMore ...", "charCount": 1051, "estimatedTokens": 263, "relevanceScore": null }
  ],
  "query": null,
  "outputFormat": "both",
  "fetchedAt": "2026-09-27T03:49:27.558Z"
}
```

| Field | How to read it |
|---|---|
| `requestedUrl`, `url`, `finalUrl`, `canonicalUrl` | The address asked for (the first two carry the same value), the one that answered after redirects, and what the page says its own address is. |
| `text` | The readable content, boilerplate removed. Absent when `outputFormat` is `markdown`. |
| `markdown` | Headings, paragraphs and lists. Absent when `outputFormat` is `json`. A page built entirely of tables falls back to the plain text. |
| `links` | Same-site links found on the page, with their anchor text, up to 250. |
| `chunks` | `chunkIndex`, `text`, `charCount`, `estimatedTokens`, and `relevanceScore` when you gave a query. With a query, the best chunks come first. |
| `wordCount`, `language`, `status` | Useful for filtering out thin pages and redirect landing pages before you embed anything. |
| `contentType` | `text/html`, or `text/markdown` for Markdown pages read through `llms.txt`. |

### 🧾 Reading the output

Page rows carry `ok: true`. Failures carry `diagnostics: true` with an `errorCode`, and are never charged.

| Row | How to spot it | Charged |
|---|---|---|
| A page | `ok` is true | yes |
| The sample row | `_sample` is true | no |
| A failed page | `diagnostics` is true, and `errorCode` says why | no |
| The site's summary | `_summary` is true; one per starting URL, after its pages, with the counts and `stopReason` | no |

| Code | What it means |
|---|---|
| `BAD_INPUT` | Not a public HTTP or HTTPS address, or an option value it cannot read (`field` names it). Credentials in the URL and other schemes land here. |
| `LEFT_OUT` | A page this crawler does not visit. The row carries no address. |
| `TIME_LIMIT` | The run stopped short of its time limit to keep what it had crawled. It still ends as succeeded, with an amber status message, even when nothing came back. |
| `OPTION_UNAVAILABLE` | Following links to other websites could not run this time, so nothing was crawled. |
| `ROBOTS_DISALLOWED` | The site's own `robots.txt` asks crawlers not to read that path. |
| `UNSUPPORTED_CONTENT` | A PDF, an image, JSON, anything that is not an HTML page. |
| `HTTP_ERROR` | The server answered with an error status. |
| `NETWORK` | The server could not be reached, or answered badly. |
| `TIMEOUT` | Slower than `requestTimeoutSecs`. |
| `RESPONSE_TOO_LARGE` | Bigger than `maxResponseSizeKb`. |
| `SSRF_BLOCKED` | The address, or one it redirected to, points inside a private network, so it was refused. |
| `UNEXPECTED_ERROR` | The run stopped early on an unexpected error. The results before this row are kept. |

One quirk of the table view: a failed row has no `url`, and the address it tried sits in `requestedUrl`, which the view does not show. Open the row or export the dataset to see which page it was.

### 💡 What people use it for

- The dull half of a RAG pipeline: crawl the docs, get chunks with token counts, embed them.
- Keeping a support bot's knowledge current by crawling the help centre again on a schedule.
- Content audits: `wordCount`, `title`, `description` and `status` across every page, in one export.
- Finding the pages that actually mention a term, using `query` and `relevanceScore`.

### 🚧 What it does not do

- **No browser.** Content a script builds after loading is not there to read. [JavaScript Website Crawler](https://apify.com/dami_studio/javascript-website-crawler) reads those pages.
- **Other websites one step deep at most**, and only when you ask.
- **HTML only.** PDFs, Word files and images come back as unsupported rather than as text.
- **Nothing behind a login.** No cookies, no sign-in, no interactive challenges.
- **1,000 pages per starting URL.** For a bigger site, give it several starting URLs, one per section.
- **No screenshots.** The screenshot generator linked below does that.
- **No summarising or tagging.** No model runs here, which is the point: nothing is invented and nothing is rephrased.
- It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.

### 🧭 Which site tool do you need?

| If you want | Use |
|---|---|
| A site as text, Markdown and chunks | This one |
| Pages that only fill in once their scripts run | [JavaScript Website Crawler](https://apify.com/dami_studio/javascript-website-crawler) |
| Pictures of the pages instead | [Website Screenshot Generator](https://apify.com/dami_studio/website-screenshot-generator) |
| A site's DNS, certificate, security headers and tech signals | [Domain Inspector](https://apify.com/dami_studio/domain-inspector) |
| The emails, phones and socials a site publishes | [Contact Details Scraper](https://apify.com/dami_studio/contact-details-scraper) |
| Full Wikipedia articles as text | [Wikipedia Scraper](https://apify.com/dami_studio/wikipedia-scraper) |
| What changed on a rival's pricing page or changelog since last week | [SaaS Competitor Monitor](https://apify.com/dami_studio/saas-competitor-change-monitor) |
| Questions and answers from Stack Overflow | [Stack Overflow Scraper](https://apify.com/dami_studio/stackexchange-scraper) |

### ❓ Questions people ask

#### Does it run JavaScript?

No. It reads the HTML the server sends, which is the whole content on most documentation and marketing sites.

#### Will it hammer my site?

It keeps to `robots.txt`, crawl delay included, between batches of pages. Drop `maxConcurrency` to 1 and it fetches one page at a time.

#### What chunk size should I use for embeddings?

Whatever your embedding model is happiest with. 1,200 characters with 120 of overlap is a reasonable default, and `estimatedTokens` on each chunk tells you where you landed.

#### Can I crawl one page only?

Set `maxDepth` to 0 and it reads the starting pages only.

#### Do I get both text and Markdown?

With `outputFormat: both`, yes, on the same row. Pick one if the rows are getting large.

#### Can I call it from code or connect it to an AI assistant?

The [API tab](https://apify.com/dami_studio/website-intelligence-crawler/api/python) has code for Python, JavaScript
and the CLI. MCP clients like Claude and ChatGPT use
`https://mcp.apify.com/?tools=fetch-actor-details,dami_studio/website-intelligence-crawler`; coding assistants can
load [our skill
file](https://api.apify.com/v2/key-value-stores/BTBHU3DdiuoF4JOsK/records/apify-web-checks-monitors-SKILL.md). All
three bill your Apify account at the same price.

#### Is crawling a public site legal?

It reads public pages the way a signed-out visitor sees them, with no account, and honours `robots.txt`. Pages can hold personal data, and the e-mails and numbers `includeContacts` returns are personal data under GDPR, PIPEDA and similar laws: keep only what you need, and check the anti-spam laws (CAN-SPAM, CASL and the EU's rules) before you write to anyone. Site terms still apply to what you do with the content afterwards. Apify's write-up on [the legality of web scraping](https://blog.apify.com/is-web-scraping-legal/) is a good starting point, and we are not lawyers.

### 🆘 If something breaks

Open the **Issues** tab on the actor page. Send the starting URL and the run ID. If a diagnostic row landed, its `errorCode` and `requestedUrl` usually name the reason already.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) URLs to crawl. Values are normalized and deduplicated.

## `urls` (type: `array`):

Additional public HTTP(S) URLs. Same-site links are followed from each seed.

## `maxPages` (type: `integer`):

Maximum successful or diagnostic page records per seed URL.

## `maxDepth` (type: `integer`):

Maximum same-site link depth from each seed. Zero fetches only the seed page.

## `maxConcurrency` (type: `integer`):

Maximum requests processed concurrently for each seed site.

## `useRobotsTxt` (type: `boolean`):

Honor public robots.txt disallow and crawl-delay rules.

## `query` (type: `string`):

Whitespace-separated terms used for deterministic ranking. No AI or API key is used.

## `chunkSizeChars` (type: `integer`):

Target maximum characters in each deterministic content chunk.

## `chunkOverlapChars` (type: `integer`):

Characters repeated between adjacent chunks to preserve context.

## `outputFormat` (type: `string`):

Keep clean text, Markdown, or both representations in each page record.

## `requestTimeoutSecs` (type: `integer`):

Maximum time allowed for each HTTP request.

## `maxResponseSizeKb` (type: `integer`):

Maximum response body size accepted from one page in kilobytes.

## `fallbackToProxy` (type: `boolean`):

Retry a page that could not be read.

## `proxyConfiguration` (type: `object`):

Optional proxy configuration. Direct traffic is always tried first.

## `maxPagesPerRun` (type: `integer`):

A page limit shared by all start URLs together. Leave it empty to keep only the limit per start URL.

## `includeUrlGlobs` (type: `array`):

Follow a link only when it matches one of these patterns. \* matches inside one folder, \*\* across folders. A pattern starting with / is matched against the path and query (/docs/\*\*), a full address against the whole URL. Start URLs are always read. Up to 20 patterns.

## `excludeUrlGlobs` (type: `array`):

Never follow a link that matches one of these patterns, for example /tag/\*\* or \*\*?page=\*. Same pattern rules as above.

## `stayUnderStartPath` (type: `boolean`):

Only follow links inside the start URL's folder: a crawl started at /docs/ stays in /docs/ and leaves /docs-old/ alone.

## `includeSubdomains` (type: `boolean`):

Treat subdomains of the start site as the same site, so a crawl of example.com also reads docs.example.com and help.example.com.

## `followNavigationLinks` (type: `boolean`):

Also follow links in menus, headers and footers. Without it only links in the body of the page are followed, which can leave a docs site with a sidebar at one page.

## `skipFileLinks` (type: `boolean`):

Don't request links that end in a file type: PDFs, office files, images, media, archives, scripts, styles and feeds.

## `countOnlyGoodPages` (type: `boolean`):

Failed, refused and left-out pages don't count toward Maximum pages. The crawl still stops after twice that many tries.

## `useSitemap` (type: `boolean`):

Also start from the pages in the site's sitemap (named in robots.txt, or /sitemap.xml), sitemap indexes included. A start URL ending in .xml is read as a sitemap.

## `useLlmsTxt` (type: `boolean`):

Also start from the pages listed in the site's llms.txt. Markdown pages it links are read as Markdown.

## `followExternalLinks` (type: `boolean`):

Also read pages on other websites that the site links to. One step away only: links on those pages are not followed. Their robots.txt is always honoured, and sign-in and checkout addresses are skipped.

## `maxPagesPerOtherSite` (type: `integer`):

How many pages to read from each other website when following links to other websites.

## `requestLanguage` (type: `string`):

The language list sent with each request, for sites that answer in the visitor's language, for example de-DE,de;q=0.9. Empty means US English.

## `markdownStyle` (type: `string`):

Basic keeps headings, paragraphs and lists. Rich also keeps links, bold and italics, tables and code blocks. Chunks are always cut from the plain text.

## `contentExtractor` (type: `string`):

Main content reads the page's main or article element, or the body when there is none. Article reader picks the block with the most paragraph text. Whole page reads the body.

## `keepSelector` (type: `string`):

Keep only the parts of the page that match this CSS selector, for example article .content. When nothing matches, the normal text is kept and keepSelectorMatched is false.

## `removeSelector` (type: `string`):

Remove the parts of the page that match this CSS selector before the text is read, for example .sidebar, .newsletter. Links are still followed from them.

## `removeCookieBanners` (type: `boolean`):

Remove cookie and consent dialogs from the text, so their wording stays out of the first chunk.

## `dropRepeatedLines` (type: `boolean`):

Drop paragraphs and other text blocks already seen on an earlier page of the same site, such as sidebars and sign-up boxes. The text is then rebuilt block by block, and repeatedLinesDropped counts what went.

## `includeHtml` (type: `boolean`):

Add the cleaned HTML of the content to each row as html. Rows get several times bigger.

## `includeContacts` (type: `boolean`):

Add the e-mail addresses and tap-to-call phone numbers the site itself publishes on its pages, as emails and phones. Not read on other websites. They are personal data in many countries: keep only what you need.

## `saveSkippedUrls` (type: `boolean`):

Save the links your rules left out, with the reason, in the key-value store record SKIPPED_URLS. Up to 10,000.

## `skipCanonicalDuplicates` (type: `boolean`):

Optional add-on. Leave out a page whose canonical address was already crawled in this run, such as copies with tracking parameters.

## `changedSince` (type: `string`):

Optional add-on for repeat crawls. A date such as 2026-10-01, or a span such as 7 days. Pages the server reports as unchanged since then are left out, and sitemap entries dated earlier are not requested. Pair it with the sitemap: a page left out is not read for links when the server sends no copy.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 1,
  "maxConcurrency": 3,
  "useRobotsTxt": true,
  "chunkSizeChars": 1200,
  "chunkOverlapChars": 120,
  "outputFormat": "both",
  "requestTimeoutSecs": 15,
  "maxResponseSizeKb": 2048,
  "fallbackToProxy": false,
  "stayUnderStartPath": false,
  "includeSubdomains": false,
  "followNavigationLinks": false,
  "skipFileLinks": false,
  "countOnlyGoodPages": false,
  "useSitemap": false,
  "useLlmsTxt": false,
  "followExternalLinks": false,
  "maxPagesPerOtherSite": 3,
  "markdownStyle": "basic",
  "contentExtractor": "default",
  "removeCookieBanners": false,
  "dropRepeatedLines": false,
  "includeHtml": false,
  "includeContacts": false,
  "saveSkippedUrls": false,
  "skipCanonicalDuplicates": false
}
```

# Actor output Schema

## `results` (type: `string`):

Page records in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("dami_studio/website-intelligence-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("dami_studio/website-intelligence-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call dami_studio/website-intelligence-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dami_studio/website-intelligence-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JYu9aJftUKpamOBdy/builds/tEzxzi0YVe8nk3aFF/openapi.json