# RAG Web Browser (`parseforge/rag-web-browser`) Actor

Fetches web pages from search queries or direct URLs and returns clean content as markdown, text, and optional HTML. Designed to feed RAG pipelines and LLMs with structured data.

- **URL**: https://apify.com/parseforge/rag-web-browser.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** AI, Automation
- **Features:** Limited permissions
- **Stats:** 20 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.00 / 1,000 result items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner-v4.webp)](https://apify.com/parseforge?fpr=vmoqkp)

### RAG Web Browser

**Turn any web page or search query into clean, structured data for your RAG pipelines and LLMs.** Fetch content as markdown, text, or HTML, with optional outbound links. No sitemaps or complex crawling rules required.

Building a RAG application or feeding an LLM requires clean, structured content, but websites are messy. This Actor fetches web pages from search queries or direct URLs and returns their content in a consistent, machine-readable format. It handles the extraction so you get markdown, plain text, or raw HTML ready for your vector database.

| Who uses it | What they scrape Web for |
|---|---|
| LLM developers | Building a knowledge base for a RAG chatbot by fetching and cleaning documentation pages. |
| Data engineers | Ingesting web content into a data pipeline for downstream processing and analysis. |
| SEO specialists | Extracting clean text content from competitor pages for content gap analysis. |
| Researchers | Collecting and formatting articles from multiple sources for literature reviews. |

### What it does

This Actor fetches web pages from a search query or a list of URLs and returns each page's content as a flat row with markdown, text, and optional HTML and outbound links.

- 🔍 **Search-based fetching:** Provide a search query and the Actor fetches the top results, up to 100 pages.
- 🔗 **Direct URL fetching:** Paste a list of specific URLs to fetch their content directly, bypassing search.
- 📝 **Multiple output formats:** Get content as markdown, plain text, and HTML in a single run for maximum flexibility.
- 🔗 **Outbound link extraction:** Optionally collect all links from each page to map the information neighborhood.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with Web data

**🤖 Populate a RAG knowledge base.**

An LLM developer runs the Actor on a set of documentation URLs to fetch clean markdown, then chunks and embeds the results into a vector store for a support chatbot.

**📊 Aggregate research from search results.**

A market analyst enters a search query about a new industry trend, fetches the top 20 articles as plain text, and runs them through a summarization model.

**🔗 Map a website's link graph.**

An SEO specialist fetches a competitor's homepage with the 'Include Links' option enabled to extract all outbound links and understand their partnership strategy.

### Why choose this scraper

|  | What you get |
|---|---|
| **Clean, structured output** | Every page is returned as markdown, text, and optionally HTML, ready for chunking and embedding. |
| **No complex setup** | Start with a simple search query or a list of URLs. No sitemaps, CSS selectors, or crawling rules needed. |
| **Built for RAG** | Designed specifically to feed Retrieval-Augmented Generation pipelines with consistently formatted documents. |

### How it compares

No other Store actor targets Web the same way, so the honest comparison is with the alternatives teams actually weigh.

| | RAG Web Browser | Build it in-house | By hand |
|---|---|---|---|
| Setup | Run it now, zero config | Days of engineering | None, but hours per pull |
| When Web changes | Maintained for you | You fix it | You re-learn the page |
| Proxies, retries, anti-bot | Built in | Your problem | Browser only |
| Output | Fixed JSON schema, CSV/Excel export | Whatever you build | Copy-paste |
| Cost | Pay per result | Engineering time | Analyst hours |

### Configure the run

Drive the Actor with a search query or a list of direct URLs, and control the number of pages fetched and the output formats returned. The Input tab lists every parameter.

A first run with the defaults:

```json
{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}
```

A larger pull:

```json
{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 200,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}
```

### Pricing

Pay-per-result: **$0.012 per result** collected. You pay only for the results written to your dataset.

| Results collected | Approximate cost |
|---|---|
| 100 results | $1.20 |
| 1,000 results | $12.00 |
| 10,000 results | $120.00 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [RAG Web Browser](https://apify.com/parseforge/rag-web-browser?fpr=vmoqkp).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to Web through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/rag-web-browser"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why is the returned text or markdown empty or incomplete?**

The page might rely on JavaScript to load content. The Actor fetches the static HTML. Try viewing the page source in your browser to see what is available without JavaScript. If the content is missing, you may need a headless browser solution.

**I'm getting fewer results than my 'Max Search Results' setting.**

The search engine may return fewer results than requested for your query. Try a more general search query or use direct URLs if you know the specific pages you need.

**Why am I hitting the 'Maximum results' limit as a free user?**

Free accounts on Apify have a limit of 10 items per run for this Actor. To process more items, upgrade to a paid plan and increase the 'Maximum results' value.

**The HTML output looks different from what I see in my browser.**

The Actor returns the raw HTML source code. Your browser applies CSS and JavaScript to render the visual page. The source code is the correct input for machine processing and content extraction.

### FAQ

| Question | Answer |
|---|---|
| What is the difference between using a search query and direct URLs? | A search query lets you find and fetch content dynamically based on a topic. Direct URLs let you specify exactly which pages to fetch, which is useful when you already know the target pages. |
| What output formats are supported? | The Actor always returns markdown. You can optionally include plain text and the raw HTML. Select the formats you need in the 'Output Formats' input field. |
| How many pages can I fetch in one run? | Free users are limited to 10 items. Paid users can fetch up to 1,000,000 items by adjusting the 'Maximum results' setting. |
| Does this Actor execute JavaScript on the pages? | The Actor fetches the initial HTML content. For pages that rely heavily on client-side JavaScript to render content, the returned HTML may not include dynamically loaded elements. |
| Can I use this to scrape a whole website? | This Actor is designed for fetching a specific set of pages from a search or a URL list. For crawling an entire website by following links, use the Website Content Crawler. |
| What is the 'Include Links' option for? | When enabled, the Actor extracts all outbound links found on each page and includes them in the output. This is useful for analyzing a page's link profile or discovering related resources. |
| How do I use the output in my RAG pipeline? | The markdown or text output is clean and ready for chunking. You can feed it directly into a text splitter and then into an embedding model to create vectors for your vector database. |
| Is there a way to filter the search results before fetching? | The Actor fetches the top N results from the search query as specified in 'Max Search Results'. It does not apply content-based filters before fetching, but you can process the output afterward. |

### Related actors

Browse the full [ParseForge collection](https://apify.com/parseforge?fpr=vmoqkp) for more scrapers.

🆘 **Need help?** Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by the respective website owners. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `query` (type: `string`):

Web search query. Alternative to Start URLs.

## `startUrls` (type: `array`):

Specific URLs to fetch. Alternative to query.

## `maxResults` (type: `integer`):

Number of search results to fetch (when using query).

## `maxItems` (type: `integer`):

Free users: Limited to 10 items (preview). Paid users: Optional, max 1,000,000

## `outputFormats` (type: `array`):

Which content formats to include. markdown is always included.

## `includeLinks` (type: `boolean`):

Include all outbound links from each page.

## Actor input object example

```json
{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}
```

# Actor output Schema

## `results` (type: `string`):

Complete dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "Anthropic Claude API pricing",
    "maxResults": 10,
    "maxItems": 10,
    "outputFormats": [
        "markdown",
        "text"
    ],
    "includeLinks": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "Anthropic Claude API pricing",
    "maxResults": 10,
    "maxItems": 10,
    "outputFormats": [
        "markdown",
        "text",
    ],
    "includeLinks": False,
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}' |
apify call parseforge/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ooY9a5K2abMjKLmjo/builds/pqIT8p1yoNgfaJhlZ/openapi.json