Website Intelligence Crawler

Website crawler for clean text, Markdown and chunks sized for an embedding model, from a public site: title, description, language, word count, the same-site links and the status code ride along on every row. No API key, no model call, nothing to authorise.

The one limit to understand first: there is no browser here. It reads the HTML the server sends, so a page that builds its content in JavaScript after loading comes back with very little text, and no setting changes that. For ordinary documentation, blogs, marketing sites and knowledge bases, the HTML is the content.

Input	Public HTTP or HTTPS URLs to start from
Output	One row per page: readable text, Markdown, chunks with token estimates, title, description, language, links, status
Ceiling	1,000 pages per starting URL, 20 levels of links
Account needed	None
Price	$0.60 per 1,000 pages ($0.0006 each) on the Free plan, a little less on paid plans. The free plan's $5 a month covers about 8,300

🔍 What Website Crawler does

From each URL you give it, it follows same-site links down to the depth you set and writes a record per page. It stays on the site it started on unless you ask it to read the websites the site links to.

You can aim it: URL patterns to follow or skip, the start folder only, subdomains, the sitemap or llms.txt as extra starting points, and one page limit for the whole run.

The text is the readable part of the page, with navigation, scripts and boilerplate stripped. The Markdown keeps the heading structure, paragraphs and lists, which is the version you want if the pages are going into a document store.

Chunks are cut to the size you ask for with a little overlap, so a sentence is not sliced in half. Each one carries its character count and an estimated token count, so you can budget an embedding run before you start it.

Give it a query and every chunk gets a relevanceScore against those words. It is plain term matching, not a model, and it is there to let you keep the 20 chunks that matter instead of all 400. The chunking and the scoring are both arithmetic, so the same input gives the same output every time.

📋 What data you get from each page

What you get	Field
The page's readable text	text
The same content as Markdown, headings kept	markdown
Chunks with character count, token estimate and relevance score	chunks
Title, description, language and canonical address	title, description, language, canonicalUrl
Same-site links with their anchor text	links
Word count and HTTP status	wordCount, status
Where it came from: the starting URL, the site, the link depth and the page that linked here	seedUrl, siteKey, depth, referrerUrl
The address asked for, and the one that answered after redirects	requestedUrl, url, finalUrl
Open Graph, site name, sharing image, published date, keywords	openGraph, siteName, imageUrl, publishedAt, keywords
Headings with their level	headings
Links to other websites with anchor text, and link totals by place on the page	externalLinks, linkStats
Robots tag, noindex and where the canonical points	indexability
Structured data: its types and the JSON-LD blocks as published	schemaTypes, jsonLd
Images with alt text and declared size	images, imageStats
Paragraph and sentence counts, reading ease, top words	contentStats
Response headers and time	responseHeaders, responseTimeMs
A flag for pages that only fill in once scripts run	needsJavaScript
Pages in the run sharing a title or description	duplicateTitleOf, duplicateDescriptionOf
Optional: cleaned HTML, published e-mails and phone numbers	html, emails, phones

▶️ How to crawl a website into text, Markdown and chunks

1. Open Website Crawler and click Try for free.
2. Put your address into Start URLs.
3. Set Maximum pages and Maximum link depth. Ten pages at depth 1 is a good look before you commit to a whole site.
4. If the crawl is feeding a search or a RAG index, set Chunk size to whatever your embedding model likes.
5. Click Start, then download the dataset as JSON, CSV or Excel, or read it from the API.

Leave the URLs empty and you get one sample row showing the shape, with nothing fetched and nothing charged.

💰 How much does it cost to crawl a website?

$0.60 per 1,000 pages, which is $0.0006 each on the Free plan. Paid plans pay a little less: $0.59 on Starter, $0.58 on Scale and $0.57 on Business. The free plan's $5 a month covers about 8,300 pages.

You pay per page that came back. The sample row is not charged, and neither is a page that failed, was disallowed by robots.txt, was too large, or turned out to be a PDF. Those still arrive as rows so you know they happened. Summary rows are free, and so are pages the two optional add-ons leave out until 9 November 2026. From that day:

Add-on	Price
skipCanonicalDuplicates, changedSince	$0.30 per 1,000 pages they fetch and leave out on the Free plan, a little less on paid plans

Fifty pages is three cents. A hundred-page site crawled weekly is about a quarter a month. If you set a maximum charge for a run, it stops at the last page that fits, and the run's status says so.

📥 What you give it

{
  "startUrls": [{"url": "https://docs.example.com/"}],
  "maxPages": 50,
  "maxDepth": 2,
  "outputFormat": "both",
  "chunkSizeChars": 1200,
  "query": "pricing plans billing"
}

Field	Default	What it is
startUrls	none	The pages to start from. Normalised and deduplicated. The Console box starts with https://example.com/.
urls	none	More starting URLs, as plain strings, if that shape suits you better.
maxPages	10	Records per starting URL, 1 to 1,000. Failed pages count too, unless countOnlyGoodPages is on.
maxDepth	1	Link depth, 0 to 20. 0 reads the starting pages only.
maxConcurrency	3	Pages fetched at once from each site, 1 to 10. Several starting URLs are also crawled side by side up to the same number, so 3 with three starting URLs can mean nine requests at once. Leave it low on small servers.
useRobotsTxt	true	Honour Disallow rules and crawl delay. The delay is kept between batches of pages, so use maxConcurrency 1 for a site that asks for one. Allow exceptions are not read, so a site that relies on them may be crawled less than it permits.
query	none	Words to score chunks against. No model, no key.
chunkSizeChars	1200	Target characters per chunk, 300 to 5,000.
chunkOverlapChars	120	Characters repeated between neighbouring chunks, up to 800. Zero behaves as 120.
outputFormat	both	json keeps the text, markdown keeps the Markdown, both keeps each.
requestTimeoutSecs	15	Per request, 3 to 60.
maxResponseSizeKb	2048	Biggest page body accepted, 128 to 8,192 KB.
fallbackToProxy	false	Retry a page that could not be read.
proxyConfiguration	none	Your own servers, if you have them.
maxPagesPerRun	none	One page limit for the whole run, across the starting URLs.
includeUrlGlobs	none	Follow only links matching these patterns, such as /docs/**. * stays inside a folder, ** crosses folders. Up to 20. Starting URLs are always read.
excludeUrlGlobs	none	Never follow links matching these patterns, such as **?page=*.
stayUnderStartPath	false	A crawl started at /docs/ stays in /docs/.
includeSubdomains	false	docs.example.com counts as part of example.com.
followNavigationLinks	false	Also follow links in menus, headers and footers.
skipFileLinks	false	Don't request links ending in a file type: PDFs, images, archives, feeds.
countOnlyGoodPages	false	Failed and left-out pages don't use up maxPages; tries stop at twice the limit.
useSitemap	false	Also start from the sitemap, indexes included. A starting URL ending in .xml is read as a sitemap.
useLlmsTxt	false	Also start from the pages in the site's llms.txt, Markdown pages read as Markdown.
followExternalLinks	false	Also read pages on websites the site links to, one step away, their robots.txt honoured, sign-in and checkout addresses skipped.
maxPagesPerOtherSite	3	Pages per other website, 1 to 20.
requestLanguage	US English	The language list sent with each request, such as de-DE,de;q=0.9.
markdownStyle	basic	rich keeps links, bold, tables and code blocks too. Chunks are always cut from the plain text.
contentExtractor	default	default reads the main or article element, article picks the block with the most paragraph text, wholePage reads the body.
keepSelector	none	Keep only what matches this CSS selector. No match keeps the normal text and sets keepSelectorMatched to false.
removeSelector	none	Remove what matches this CSS selector before the text is read.
removeCookieBanners	false	Remove cookie and consent dialogs from the text.
dropRepeatedLines	false	Drop text blocks an earlier page of the same site already had; the text is rebuilt block by block.
includeHtml	false	Add the cleaned HTML of the content as html. Rows get several times bigger.
includeContacts	false	Add the e-mails and tap-to-call numbers the site publishes, as emails and phones. Not read on other websites.
saveSkippedUrls	false	Save the links your rules left out, with the reason, in the key-value record SKIPPED_URLS.
skipCanonicalDuplicates	false	Optional add-on: leave out a page whose canonical address was already crawled in the run.
changedSince	none	Optional add-on for repeat crawls: 2026-10-01 or 7 days. Pages the server reports unchanged are left out and sitemap entries dated earlier are not requested. Pair it with useSitemap, since an unchanged page sent without a copy is not read for links.

📤 What you get back

A real row from a run on 27 September 2026, the first of five pages read from python.org. The text, the Markdown and the chunk texts are cut short, and the links list is cut to two of fifteen. No query was set, so relevanceScore is null.

{
  "ok": true,
  "requestedUrl": "https://python.org/",
  "url": "https://python.org/",
  "seedUrl": "https://python.org/",
  "siteKey": "python.org",
  "depth": 0,
  "status": 200,
  "title": "Welcome to Python.org",
  "description": "The official home of the Python Programming Language",
  "language": "en",
  "canonicalUrl": "https://python.org/",
  "text": "Get Started\nWhether you're new to programming or an experienced developer, it's easy to learn and use Python. ...",
  "markdown": "## Get Started\n\nWhether you're new to programming or an experienced developer, it's easy to learn and use Python.\n\n...",
  "links": [
    { "url": "https://python.org/about/gettingstarted/", "text": "Start with our Beginner’s Guide" },
    { "url": "https://python.org/downloads/release/python-3147/", "text": "Python 3.14.7" }
  ],
  "wordCount": 295,
  "contentType": "text/html",
  "chunks": [
    { "chunkIndex": 0, "text": "Get Started\nWhether you're new to programming ...", "charCount": 1200, "estimatedTokens": 300, "relevanceScore": null },
    { "chunkIndex": 1, "text": "26\n\nSuccess Stories\nMore ...", "charCount": 1051, "estimatedTokens": 263, "relevanceScore": null }
  ],
  "query": null,
  "outputFormat": "both",
  "fetchedAt": "2026-09-27T03:49:27.558Z"
}

Field	How to read it
requestedUrl, url, finalUrl, canonicalUrl	The address asked for (the first two carry the same value), the one that answered after redirects, and what the page says its own address is.
text	The readable content, boilerplate removed. Absent when outputFormat is markdown.
markdown	Headings, paragraphs and lists. Absent when outputFormat is json. A page built entirely of tables falls back to the plain text.
links	Same-site links found on the page, with their anchor text, up to 250.
chunks	chunkIndex, text, charCount, estimatedTokens, and relevanceScore when you gave a query. With a query, the best chunks come first.
wordCount, language, status	Useful for filtering out thin pages and redirect landing pages before you embed anything.
contentType	text/html, or text/markdown for Markdown pages read through llms.txt.

🧾 Reading the output

Page rows carry ok: true. Failures carry diagnostics: true with an errorCode, and are never charged.

Row	How to spot it	Charged
A page	ok is true	yes
The sample row	_sample is true	no
A failed page	diagnostics is true, and errorCode says why	no
The site's summary	_summary is true; one per starting URL, after its pages, with the counts and stopReason	no

Code	What it means
BAD_INPUT	Not a public HTTP or HTTPS address, or an option value it cannot read (field names it). Credentials in the URL and other schemes land here.
LEFT_OUT	A page this crawler does not visit. The row carries no address.
TIME_LIMIT	The run stopped short of its time limit to keep what it had crawled. It still ends as succeeded, with an amber status message, even when nothing came back.
OPTION_UNAVAILABLE	Following links to other websites could not run this time, so nothing was crawled.
ROBOTS_DISALLOWED	The site's own robots.txt asks crawlers not to read that path.
UNSUPPORTED_CONTENT	A PDF, an image, JSON, anything that is not an HTML page.
HTTP_ERROR	The server answered with an error status.
NETWORK	The server could not be reached, or answered badly.
TIMEOUT	Slower than requestTimeoutSecs.
RESPONSE_TOO_LARGE	Bigger than maxResponseSizeKb.
SSRF_BLOCKED	The address, or one it redirected to, points inside a private network, so it was refused.
UNEXPECTED_ERROR	The run stopped early on an unexpected error. The results before this row are kept.

One quirk of the table view: a failed row has no url, and the address it tried sits in requestedUrl, which the view does not show. Open the row or export the dataset to see which page it was.

💡 What people use it for

• The dull half of a RAG pipeline: crawl the docs, get chunks with token counts, embed them.
• Keeping a support bot's knowledge current by crawling the help centre again on a schedule.
• Content audits: wordCount, title, description and status across every page, in one export.
• Finding the pages that actually mention a term, using query and relevanceScore.

🚧 What it does not do

• No browser. Content a script builds after loading is not there to read. JavaScript Website Crawler reads those pages.
• Other websites one step deep at most, and only when you ask.
• HTML only. PDFs, Word files and images come back as unsupported rather than as text.
• Nothing behind a login. No cookies, no sign-in, no interactive challenges.
• 1,000 pages per starting URL. For a bigger site, give it several starting URLs, one per section.
• No screenshots. The screenshot generator linked below does that.
• No summarising or tagging. No model runs here, which is the point: nothing is invented and nothing is rephrased.
• It is not an official tool of any platform or company named on this page. Dami's Studio is independent and is not affiliated with or endorsed by any of them.

🧭 Which site tool do you need?

If you want	Use
A site as text, Markdown and chunks	This one
Pages that only fill in once their scripts run	JavaScript Website Crawler
Pictures of the pages instead	Website Screenshot Generator
A site's DNS, certificate, security headers and tech signals	Domain Inspector
The emails, phones and socials a site publishes	Contact Details Scraper
Full Wikipedia articles as text	Wikipedia Scraper
What changed on a rival's pricing page or changelog since last week	SaaS Competitor Monitor
Questions and answers from Stack Overflow	Stack Overflow Scraper

❓ Questions people ask

Does it run JavaScript?

No. It reads the HTML the server sends, which is the whole content on most documentation and marketing sites.

Will it hammer my site?

It keeps to robots.txt, crawl delay included, between batches of pages. Drop maxConcurrency to 1 and it fetches one page at a time.

What chunk size should I use for embeddings?

Whatever your embedding model is happiest with. 1,200 characters with 120 of overlap is a reasonable default, and estimatedTokens on each chunk tells you where you landed.

Can I crawl one page only?

Set maxDepth to 0 and it reads the starting pages only.

Do I get both text and Markdown?

With outputFormat: both, yes, on the same row. Pick one if the rows are getting large.

Can I call it from code or connect it to an AI assistant?

The API tab has code for Python, JavaScript and the CLI. MCP clients like Claude and ChatGPT use https://mcp.apify.com/?tools=fetch-actor-details,dami_studio/website-intelligence-crawler; coding assistants can load our skill file. All three bill your Apify account at the same price.

Is crawling a public site legal?

It reads public pages the way a signed-out visitor sees them, with no account, and honours robots.txt. Pages can hold personal data, and the e-mails and numbers includeContacts returns are personal data under GDPR, PIPEDA and similar laws: keep only what you need, and check the anti-spam laws (CAN-SPAM, CASL and the EU's rules) before you write to anyone. Site terms still apply to what you do with the content afterwards. Apify's write-up on the legality of web scraping is a good starting point, and we are not lawyers.

🆘 If something breaks

Open the Issues tab on the actor page. Send the starting URL and the run ID. If a diagnostic row landed, its errorCode and requestedUrl usually name the reason already.