# Website Content Crawler | Markdown for RAG (`xtech/news-source-crawler`) Actor

Turn public news, blog, and documentation pages into clean Markdown or plain text. Discover pages through sitemaps and feeds, or extract known URLs directly. Export content, metadata, and change-tracking fields for RAG and monitoring, while respecting robots.txt and website boundaries.

- **URL**: https://apify.com/xtech/news-source-crawler.md
- **Developed by:** [Xtech](https://apify.com/xtech) (community)
- **Categories:** News, SEO tools, Social media
- **Features:** Limited permissions
- **Stats:** 446 total users, 14 monthly users, 100.0% runs succeeded, 16 bookmarks
- **User rating**: 4.76 out of 5 stars

## Pricing

from $1.56 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content Crawler | Markdown for RAG

Build a content collection from public newsrooms, blogs and documentation sites. Supply known page URLs for direct extraction, or discover pages through sitemaps and feeds. Export Markdown or plain text with page metadata and content hashes for your search index, RAG pipeline or content-monitoring workflow. The crawler respects robots.txt and stays within the supplied website host. It does not sign in or bypass access controls.

**Best first run:** [open the Website Content Crawler](https://apify.com/xtech/news-source-crawler), start with one public documentation page and a $0.01 cap, then expand to sitemap or feed discovery after confirming the Markdown fits your RAG pipeline.

### Workflow: website content to Markdown for RAG

1. Open the published direct-extraction example and inspect one public documentation page. Keep the page limit at one and the example run cap at $0.01.
2. Confirm the Markdown contains the content you need, then replace the URL with a permitted page from your knowledge source. Use sitemap/feed discovery when you need multiple pages.
3. Export content and canonical URLs to your own indexing pipeline. Use content hashes to compare snapshots before deciding what to re-index. This Actor prepares content; it does not create embeddings or host a vector database.

Pricing is approximately $1.5625 per 1,000 successful pages, plus $0.00005 per start event per GB of memory (minimum one). Check the Pricing tab for the current rate.

### Best for

- Building a clean, permission-aware knowledge base from a public newsroom, blog, or documentation site
- Extracting a known set of pages cheaply with direct URL mode
- Monitoring content changes using `contentHash`, ETags, and Last-Modified values

### What it is for

- Build a reliable content library from websites and publisher sources.
- Monitor a knowledge base, blog, newsroom, documentation hub, or resource center.
- Feed clean website content into search, RAG, analysis, or editorial workflows.
- Track content changes with content hashes, ETags, and Last-Modified values.
- Collect only a section of a site, such as `/news`, `/guides`, or `/research`.

### What it does

- Starts with published sitemaps and RSS/Atom feeds to find the right pages efficiently.
- Can fetch only the supplied URLs when direct URL mode is selected, avoiding discovery requests for fast, low-cost page extraction.
- Stays on the same website host you supplied.
- Respects each website’s robots.txt rules before requesting content.
- Keeps successful content pages separate from skipped URLs and source issues.
- Gives every saved page clean content, title, page metadata, canonical URL, fetch date, and change-tracking details.
- Works with public pages only. It does not sign in, use proxies, or try to work around access controls.

### Get started

1. Add one or more website or section URLs.
2. Set the maximum number of content pages you need.
3. Optionally include or exclude paths to focus the crawl.
4. Choose site discovery or direct URL mode.
5. Choose Markdown or plain text, then run the Actor.

For a small first run, extract this public documentation page directly:

```json
{
  "sourceUrls": [{ "url": "https://docs.apify.com/actors/development/actor-definition/actor-json" }],
  "discoveryMode": "direct",
  "maxPages": 1,
  "maxConcurrency": 1,
  "contentFormat": "markdown"
}
```

This input returned useful content in a validation run; live pages can change. Keep `maxPages` small while checking the content you need. The following discovery example is a template: replace `example.com/guides` with your own public section URL.

For API clients, start with the same one-page request and set a maximum run charge in the Console before expanding the crawl:

```bash
curl -X POST "https://api.apify.com/v2/acts/xtech~news-source-crawler/runs?token=$APIFY_TOKEN&maxTotalChargeUsd=0.01" \
  -H "Content-Type: application/json" \
  -d '{"sourceUrls":[{"url":"https://docs.apify.com/actors/development/actor-definition/actor-json"}],"discoveryMode":"direct","maxPages":1,"maxConcurrency":1,"contentFormat":"markdown"}'
```

After confirming the first page, move to a bounded monitoring run by changing
only the source URL and page limit:

```json
{
  "sourceUrls": [{ "url": "https://example.com/docs" }],
  "includePaths": ["/docs"],
  "maxPages": 25,
  "contentFormat": "markdown"
}
```

This two-step workflow keeps the first paid run cheap and makes the next
scheduled or API run repeatable.

For example, to collect a website’s guide pages:

```json
{
  "sourceUrls": [{ "url": "https://example.com/guides" }],
  "includePaths": ["/guides"],
  "excludePaths": ["/guides/archive"],
  "maxPages": 200,
  "contentFormat": "markdown"
}
```

For a known article, use direct URL mode to skip sitemap and feed discovery:

```json
{
  "sourceUrls": [{ "url": "https://blog.apify.com/introducing-x402-agentic-payments/" }],
  "discoveryMode": "direct",
  "maxPages": 1,
  "contentFormat": "markdown"
}
```

For a newsroom section with a small first pass:

```json
{
  "sourceUrls": [{ "url": "https://example.com/news" }],
  "includePaths": ["/news"],
  "excludePaths": ["/news/archive"],
  "maxPages": 25,
  "contentFormat": "markdown"
}
```

### What you get

Each saved row is one clean public content page. It includes:

- Page URL and canonical URL
- Title and useful page metadata
- Markdown or plain-text content
- Word count and a content hash for change detection
- ETag and Last-Modified values when the website provides them
- Fetch timestamp and discovery details, including whether it came from a sitemap, feed, start page, or supplied URL

The default dataset contains only successful content pages. The separate **Source results** dataset explains skipped pages, robots.txt decisions, path filters, and source errors, so your usable export stays clean.

#### Example output

```json
{
  "schemaVersion": 1,
  "recordType": "content-page",
  "url": "https://example.com/news/launch",
  "canonicalUrl": "https://example.com/news/launch",
  "title": "Product launch",
  "contentFormat": "markdown",
  "content": "# Product launch\n\nClean page content...",
  "wordCount": 842,
  "contentHash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "etag": null,
  "lastModified": null,
  "fetchedAt": "2026-08-13T12:00:00.000Z",
  "provenance": {
    "sourceRootUrl": "https://example.com/news",
    "discoveryMethod": "sitemap",
    "discoveryUrl": "https://example.com/sitemap.xml",
    "requestedUrl": "https://example.com/news/launch",
    "finalUrl": "https://example.com/news/launch",
    "robotsStatus": "ALLOWED",
    "contentType": "text/html"
  }
}
```

### Tips

- Add a sitemap URL when you know it for the most complete coverage.
- Use a section URL and an include path to focus on a single topic area.
- For recurring monitoring, schedule the Actor and compare `contentHash`, `etag`, or `lastModified` between runs.
- Start with a small page limit when trying a new source, then raise it once you are happy with the coverage.

### Troubleshooting

- **No pages saved:** inspect the Source results dataset for robots, path-filter, HTTP, or content-type decisions.
- **A known page is missing:** use `discoveryMode: "direct"` with that URL and check whether the source permits it in robots.txt.
- **The crawl is broader than expected:** set `includePaths` and `excludePaths`, and start with a low `maxPages` value.
- **The output is not suitable for your pipeline:** choose `contentFormat: "markdown"` for publishing/RAG or `"text"` for simpler search and analysis.

### Related research workflows

- [Google Ads Transparency Scraper](https://apify.com/xtech/google-ad-transparency-scraper) — compare public ad activity with the content a brand publishes.
- [Google Keyword Scraper Pro](https://apify.com/xtech/google-keyword-scraper-pro) — compare search interest with news coverage.

### Pricing and run budget

At the current base price, one saved content page costs **$0.001562536477** (about **$1.56 per 1,000 pages**). A run-start fee of **$0.00005 per allocated GB of memory** applies, with a minimum of one event. Discounts, when available, are shown in the Console.

Start with one direct page and a **$0.01 maximum run charge**. Set the spending limit in the Console or pass `maxTotalChargeUsd` when starting through the API. Skipped pages, robots.txt decisions, and source errors are kept in Source results rather than billed as content pages.

# Actor input Schema

## `sourceUrls` (type: `array`):

Required. Add up to 20 public website or section URLs. Use a section URL when you only need content from that area of a site.

## `sitemapUrls` (type: `array`):

Add a sitemap when you know it. It must belong to one of the website hosts above. The crawler also looks for the website’s published sitemap and feed links automatically.

## `discoveryMode` (type: `string`):

Choose site discovery for broad coverage, or direct URL mode when sourceUrls already contain the exact pages you need. Direct URL mode skips robots sitemap/feed/home-page discovery and is faster and cheaper for known pages.

## `maxPages` (type: `integer`):

Maximum number of clean content pages to save in this run.

## `includePaths` (type: `array`):

Only save pages below these website paths, for example /news or /guides. Leave empty to include all paths on each supplied host.

## `excludePaths` (type: `array`):

Skip pages below these website paths, for example /tags or /archive.

## `contentFormat` (type: `string`):

Choose clean Markdown for publishing and AI workflows, or plain text for simple analysis and search.

## `maxConcurrency` (type: `integer`):

How many pages to review at once. Use a lower number for a gentler run across several websites.

## Actor input object example

```json
{
  "sourceUrls": [
    {
      "url": "https://blog.apify.com/"
    }
  ],
  "discoveryMode": "discover",
  "maxPages": 100,
  "excludePaths": [
    "/archive"
  ],
  "contentFormat": "markdown",
  "maxConcurrency": 4
}
```

# Actor output Schema

## `contentPages` (type: `string`):

Successful public content pages from the default dataset.

## `sourceResults` (type: `string`):

Separate unbilled results for skipped pages, source limits, and access outcomes.

## `runSummary` (type: `string`):

A compact summary of discovery and saved content pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sourceUrls": [
        {
            "url": "https://blog.apify.com/"
        }
    ],
    "excludePaths": [
        "/archive"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("xtech/news-source-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sourceUrls": [{ "url": "https://blog.apify.com/" }],
    "excludePaths": ["/archive"],
}

# Run the Actor and wait for it to finish
run = client.actor("xtech/news-source-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sourceUrls": [
    {
      "url": "https://blog.apify.com/"
    }
  ],
  "excludePaths": [
    "/archive"
  ]
}' |
apify call xtech/news-source-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,xtech/news-source-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cFZNFC2dBBtuyslrE/builds/J3euDNUbLXSqoTWf6/openapi.json