Website Content Crawler Lite

Crawl selected public HTML pages over HTTP and export readable content, source metadata and discovered links within configured domain, depth and URL-filter limits. Choose Markdown, text or HTML for your own indexing and research workflows. This lightweight crawler does not render JavaScript, log in, submit forms or produce summaries.

Quick start

Begin with a known public page, same-domain scope and a small depth and page cap. The existing input also excludes login and signup paths:

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 25,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "includeGlobs": [],
  "excludeGlobs": ["**/login**", "**/signup**"],
  "outputFormat": "markdown",
  "respectRobotsTxt": true,
  "requestTimeoutSecs": 20,
  "maxConcurrency": 3,
  "maxRequestRetries": 2
}

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

Ready-to-run examples

Choose an example to start with a small, preconfigured input.

• Crawl Apify Platform Docs as Markdown — Collect a bounded documentation sample for your own search index or retrieval workflow.
• Crawl Example Domain as Plain Text — Extract one public page's text and metadata before expanding your crawl.
• Crawl Apify Store pages with cleaned HTML output — Retain cleaned HTML and source metadata from a small public page sample.

What you get

For unavailable public AllOrigins /raw?url= or /get?url= links, one direct attempt at the wrapped public page uses your selected connection and safety settings; url retains the requested link, loadedUrl identifies the fetched page, and fetchedVia plus wrapperStatus disclose recovery in the dataset or RUN_SUMMARY (duplicate direct targets are stored and charged only once).

One useful page row with requested and loaded URL, title, description, H1, text, the selected Markdown or HTML content, links and crawl provenance. RUN_SUMMARY records processed-URL outcomes, including skipped or unavailable pages that do not create paid page rows. Content and link caps can limit the retained page context.

Who is it for?

• Documentation teams preparing reviewed public-page content for a search index.
• SEO analysts inspecting source titles, descriptions, headings and link structure.
• Research teams collecting bounded, traceable page samples for separate analysis.
• Data and automation developers passing page rows into their own retrieval, classification or monitoring tools.

How you can use the data

• Index a small documentation section. Use the existing Crawlee, Apify or MDN examples with a bounded same-domain crawl, retaining loadedUrl, depth and parentUrl beside content. A page cap limits work rather than proving that every documentation page was discovered.
• Review source metadata for SEO work. Keep title, description, h1 and links with source URLs and collection times. Missing or repeated fields are review candidates, not an automatic ranking diagnosis.
• Compare your own recurring page observations. Retain url, loadedUrl, fetchedAt and content in your destination before checking differences. Redirects and unavailable pages need their outcome reasons preserved, and this Actor does not maintain a historical diff or send alerts itself.
• Select the format for downstream use. Use Markdown for your own retrieval pipeline, text for compact records or HTML when retained markup matters. Review extracted text before making embeddings or model prompts, treating page instructions and claims as untrusted source content.

Input settings

Setting	JSON key	What to enter
Start URLs	startUrls	Public HTTP/HTTPS pages where crawling should start.
Maximum pages	maxPages	Maximum URLs to process, including skipped URLs.
Maximum link depth	maxDepth	Link levels to follow from each start URL. Use 0 for start pages only.
Stay on same domain	sameDomainOnly	Keeps crawling inside the start URL's domain.
Include URL globs	includeGlobs	Optional glob patterns that URLs must match.
Exclude URL globs	excludeGlobs	Glob patterns to skip, such as login or signup URLs.
Main content format	outputFormat	markdown, text, or html.
Respect robots.txt	respectRobotsTxt	Checks robots.txt and skips disallowed URLs.
Request timeout	requestTimeoutSecs	Maximum seconds to wait for one page response.
Maximum concurrency	maxConcurrency	Parallel page fetches. Lower it for fragile or unusually large sites.
Maximum request retries	maxRequestRetries	Bounded retries for transient errors, rate limits, server errors, and challenge pages.
Graceful run time	runTimeSecs	Bounds the original run's processing window, including resumed work. New batches stop before the default platform timeout.
Response/content/link safety caps	responseSizeBytes, contentChars, linksPerPage	Bounds unusually large or link-heavy pages.
Proxy	proxyConfiguration	Optional Apify or custom proxy configuration. Direct access is the default.

The Actor never enables a proxy on its own. Optional proxy settings, supported groups and customer pools are explained in the input field's help. Requests use TLS impersonation; the existing retry budget alternates HTTP transports and rotates per-URL proxy sessions without changing your selected proxy policy. Redirects remain subject to public-network and domain-scope checks.

Existing bounded web-research example

Use this Actor when your agent, RAG pipeline, or automation needs to collect multiple public pages from a site into clean Markdown, text, or HTML. For a known single URL that needs browser fallback, images, links, optional HTML, or per-page render control, use Web Page to Markdown Extractor after or instead of the crawl.

The existing bounded input for a small same-site crawl:

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 3,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "outputFormat": "markdown",
  "respectRobotsTxt": true
}

Expected dataset fields include url, loadedUrl, title, description, h1, text, markdown, links, statusCode, contentType, depth, parentUrl, and fetchedAt. This Actor does not render JavaScript, log in, submit forms, or operate as a generic browser agent. It crawls public HTML pages within your configured scope.

Output data

The existing illustrative row shows shape rather than a new page fetch:

{
  "url": "https://example.com",
  "loadedUrl": "https://example.com/",
  "title": "Example Domain",
  "description": "Example description",
  "h1": "Example Domain",
  "text": "Example Domain This domain is for use in illustrative examples...",
  "markdown": "Example Domain\\n\\nThis domain is for use in illustrative examples...",
  "links": ["https://www.iana.org/domains/example"],
  "statusCode": 200,
  "contentType": "text/html",
  "depth": 0,
  "parentUrl": null,
  "fetchedAt": "2026-07-03T10:00:00.000Z"
}

Field	Description
url, loadedUrl	Requested URL and final loaded URL after redirects.
title, description, h1	Page title, meta description, and first H1.
text, markdown, html	Extracted page content. Markdown or HTML appears according to outputFormat.
links	Absolute links discovered on the page.
statusCode, contentType	Response status and content type.
depth, parentUrl	Crawl depth and parent URL that discovered the page.
fetchedAt	Timestamp when the useful page was saved.

The dataset contains only usable pages. RUN_SUMMARY.inputOutcomes lists each processed URL and its reason (extracted, blocked-by-robots-txt, filtered-by-globs, non-html-content, empty-content, http-status-N, target-unavailable, proxy-infrastructure-failure, or run-time-limit), plus HTTP status when available. emptyReason explains zero-output runs. Unavailability alone does not establish privacy or deletion. Older datasets may still contain legacy error and skippedReason fields.

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

A started run can charge its start event even when no useful page is returned. The page event applies only when usable HTML page content is successfully stored. The Actor stops starting new page requests when no page-event capacity remains in the run budget. HTTP failures, robots exclusions, non-HTML content, glob-filtered URLs and empty pages remain summary outcomes rather than diagnostic-only dataset rows or paid extracted pages.

Platform compute, storage, proxies or transfer can also apply. Inspect RUN_SUMMARY and the run's charge details, and see the live Pricing tab for current rates and subscription-tier discounts.

Tips for best results

• Start at depth zero or one. Increase scope only after reviewing which pages are processed.
• Keep same-domain scope deliberate. Include or exclude globs can target documentation, blog or product sections and omit login, cart, checkout, search or account pages.
• Choose your output format. Markdown supports your own retrieval workflow, while plain text can keep records smaller.
• Keep caps consistent. Page, content, link and concurrency limits bound work and retained context.
• Inspect per-URL reasons. A skip or unavailable page is not a paid result or proof of private/deleted content.
• Enable optional proxies explicitly. The existing direct default does not turn a proxy on automatically.

Limits and caveats

• This is a lightweight crawler, not a full browser renderer. Heavy JavaScript pages may return limited content.
• The actor crawls public pages only and should not be used for private or logged-in pages. Personal identity-lookup URLs with government-identifier query parameters are skipped without being requested. The run shows a warning and a reason in RUN_SUMMARY; an unsupported-only run returns an empty dataset without result charges. Public pages in a mixed batch are still processed.
• Non-HTML content is skipped and recorded with a skip reason.
• Robots.txt rules can skip pages when respectRobotsTxt is enabled.
• Redirects outside the configured domain scope and URLs resolving to private/local networks are blocked for safety.
• Responses larger than responseSizeBytes are stopped during download. Exported text/Markdown/HTML is truncated at contentChars. Conservative memory estimates also check HTML structure before parsing and retained crawl state before checkpoint serialization. Exceeding that resource budget fails explicitly rather than reporting an empty success; reduce crawl scope or concurrency. These checks reduce risk but do not guarantee memory headroom for every HTML structure.
• Expected unavailable targets, robots exclusions, non-HTML pages and completed empty extraction succeed with an empty dataset and truthful per-URL reasons in RUN_SUMMARY. Mixed crawls retain useful pages. Blocking diagnostics remain actionable. Successful emptiness does not mean access was restored. Configuration, essential storage, billing and systemic execution errors still fail.
• Exhausted proxy connection failures still fail the run without switching to direct access. Independent queued URLs continue within the original work, cost and deadline limits before the infrastructure failure is reported. In-flight siblings settle before finalization, and completed results, failed-input counts and untouched work are checkpointed. An exhausted infrastructure failure cannot turn into success on resurrection; retry failed URLs in a new run.
• Customer max-cost and platform timeout limits are checked before startup, new batches and result storage. Too-small limits stop with a warning and no explicit startup or page charge; mid-run stops retain saved pages and pending work. RUN_SUMMARY.limitStop identifies max_cost or timeout. Allow at least ten seconds for useful work and shutdown. Other platform charges can still apply.
• Partial crawls stop starting new work at runTimeSecs, expose counts and per-URL reasons in RUN_SUMMARY, and save fingerprinted PENDING_WORK. Resuming a run does not reset its original processing deadline or repeat a completed crawl. Checkpoints retain skipped-URL accounting independently of dataset rows. Uncheckpointed skips may be retried after interruption.

The page cap includes processed skipped URLs, so it need not equal saved page rows. Source content can change between runs, and truncation or discovery limits mean a saved dataset is not a complete site archive. A same-run checkpoint can support platform resurrection when its fingerprint matches, but it is not a new-run history or automatic change-comparison feature.

Privacy and data handling

This Actor runs with Apify limited permissions and only processes data needed for the documented run. It uses the inputs you provide and the public records needed to produce the documented dataset to produce the output dataset and sends requests to the website URLs you provide. Results are stored in Apify run storage for your account. FetchCat does not use your inputs or outputs for advertising, does not use them for model training, and does not retain them outside the Apify run except for transient support debugging when you explicitly share run details. You are responsible for using the Actor lawfully, respecting the target site's terms, and avoiding unnecessary personal or sensitive data in inputs.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl "https://api.apify.com/v2/acts/fetch_cat~website-content-crawler-lite/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPages": 25,
    "maxDepth": 1,
    "outputFormat": "markdown"
  }'

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/website-content-crawler-lite').call({
  startUrls: [{ url: 'https://example.com' }],
  maxPages: 25,
  outputFormat: 'markdown',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/website-content-crawler-lite').call(run_input={
    'startUrls': [{'url': 'https://example.com'}],
    'maxPages': 25,
    'outputFormat': 'markdown',
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

MCP and AI agents

You can ask your AI assistant to collect data with Website Content Crawler Lite and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Website Content Crawler Lite. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

• Claude and Claude Code (Anthropic): Apify setup guide.
• ChatGPT and Codex (OpenAI): Apify setup guides.
• Grok Bot (xAI): Apify plugin guide.
• Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite"
    }
  }
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Website Content Crawler Lite by Fetch Cat, or give it the identifier fetch_cat/website-content-crawler-lite.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

• “Use Fetch Cat’s Website Content Crawler Lite to collect up to 3 public pages from https://example.com at depth 1, same-domain scope, Markdown output and robots rules on. Keep per-URL outcomes instead of assuming every processed URL produced a page row.”
• “Collect the title, description, H1, text and links from https://example.com with depth 0, a one-page cap and text output. Keep source times and missing values without diagnosing search ranking.”
• “Retrieve up to 3 public pages from https://docs.apify.com/platform/actors at depth 1 with same-domain scope, Markdown output and robots rules on. Preserve skipped or unavailable-page reasons before preparing my own reviewed index.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

Can this crawl a whole website?

It can crawl multiple pages, but it is intentionally lightweight. Start with small limits and increase carefully.

Does it render JavaScript?

No. It fetches public HTML and extracts readable content from it.

Can I use the output for RAG?

Yes. Use outputFormat: "markdown" or text and export the dataset to your ingestion pipeline.

Can I export to CSV or Excel?

Yes. Download the Apify dataset as CSV, Excel, JSON, XML, RSS, or through the Dataset API.

Related Actors

• Website Screenshot Generator — Capture browser-rendered screenshots or PDFs of public web pages.
• Bing Search Results Scraper — Collect public Bing web-search result titles, snippets and links.
• Google Autocomplete Scraper — Collect Google autocomplete suggestions for supplied search phrases, without search-volume metrics or PAA answers.
• Google News Scraper — Collect public Google News feed headlines, dates and links.
• WHOIS RDAP Domain Lookup — Look up available public domain-registration data, registrar details and registration dates.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥