# RAG Web Browser — Web Content for AI & LLMs (`nexgendata/rag-web-browser`) Actor

Web browser for RAG pipelines and AI agents. Searches DuckDuckGo, scrapes the top results, returns clean Markdown with citation URLs. Feed your LLM with real-time web data. Works with Claude, GPT, LangChain, CrewAI. No API key needed.

- **URL**: https://apify.com/nexgendata/rag-web-browser.md
- **Developed by:** [NexGenData](https://apify.com/nexgendata) (community)
- **Categories:** AI, Agents, Developer tools
- **Features:** Limited permissions
- **Stats:** 17 total users, 1 monthly users, 90.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 web pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Web Browser

RAG Web Browser is a NexGenData tool on Apify that returns clean Markdown of top search results with citation URLs from DuckDuckGo search results and the pages they link to.

Web Content for AI & LLMs

> Public web content, RAG-ready.

### 📊 Sample Output

![3 real rows delivered by RAG Web Browser — Web Content for AI & LLMs — run mzd1reqGkenJ2bnlD on build 0.0.59](https://api.apify.com/v2/key-value-stores/8gLgXMBveEI1tTz1z/records/rag-web-browser-sample)

| `url` | `title` | `description` | `markdown_content` | `content_length` | `scraped_at` |
|---|---|---|---|---|---|
| https://lushbinary.com/blog/rag-retrieval-augmented-generation-produc… | RAG Production Guide 2026: Retrieval-Augmented Generation | Lushbinary | Complete RAG production guide: hybrid search, agentic RAG, reranking,… | ## One Subscription. Every Flagship AI Model. Stop juggling multiple… | 173 | 2026-09-25T04:04:46.428037+00:00 |
| https://datarmatics.com/how-to-build-rag-pipeline-guide-2026/ | How to Build a RAG Pipeline: Complete Guide for 2026 | Datarmatics | Learn how to build a RAG pipeline from scratch — chunking, embeddings… | Retrieval-Augmented Generation (RAG) has become the default architect… | 10021 | 2026-09-25T04:04:48.344719+00:00 |
| https://aipromptshub.co/blog/rag-pipeline-architecture-best-practices | RAG Pipeline Architecture Best Practices (2026): The Complete Enginee… | A sourced, model-specific guide to RAG pipeline architecture best pra… | ## 8. Generation Model Selection: Matching Model Capability to RAG U… | 2636 | 2026-09-25T04:04:49.512400+00:00 |

Real rows from run `mzd1reqGkenJ2bnlD` on build 0.0.59 (2026-09-25), unedited apart from masked emails/phones and shortened long text; fields the source does not publish are empty.

### ⚙️ Sample inputs

This is the Store example input — the same input Apify's automated check runs — so it is proven to return rows on the current build. Paste it into the input form, or send it through the API, CLI or a schedule:

```json
{
  "query": "best practices for RAG pipelines 2026",
  "maxResults": 3,
  "includeMetadata": true,
  "maxContentLength": 10000
}
```

### 🔧 Input reference

| Field | Type | Default | What it does |
|---|---|---|---|
| `query` | string | `"best practices for RAG pipelines 2026"` | A search query (the actor searches and reads top results) or a direct URL to read. |
| `maxResults` | integer | `3` | Maximum pages to fetch and return. Defaults to 5 (evaluation-sized); raise it for full runs (each result is billed). |
| `maxContentLength` | integer | `10000` | Maximum characters of cleaned content to return per page. |
| `includeMetadata` | boolean | `true` | If enabled, includes page metadata (title, description, etc.). |

### 🧾 JSON sample record

One real record from run `mzd1reqGkenJ2bnlD` (emails/phones masked, long text shortened):

```json
{
  "url": "https://lushbinary.com/blog/rag-retrieval-augmented-generation-production-guide/",
  "title": "RAG Production Guide 2026: Retrieval-Augmented Generation | Lushbinary",
  "description": "Complete RAG production guide: hybrid search, agentic RAG, reranking, chunking strategies, vector DBs (Pinecone, Weaviate), and RAGAS evaluation. Updated April 2026.",
  "markdown_content": "## One Subscription. Every Flagship AI Model.\n\nStop juggling multiple AI subscriptions. WidelAI gives you access to Claude, GPT, Gemini, and more - all under a single plan.",
  "content_length": 173,
  "scraped_at": "2026-09-25T04:04:46.428037+00:00"
}
```

### 💰 Pricing

| Event | Price (USD) | When it is charged |
|---|---|---|
| Actor Start (`apify-actor-start`) | $0.005 | Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). |
| Web page (`apify-default-dataset-item`) | $0.01 | Charge per web page extracted and converted to Markdown |

Pay-per-event: you pay only for what the run delivers. A run that delivers nothing bills no result events (only the actor-start event, when the actor defines one). Example: a run that delivers 100 results costs 100 × $0.01 = $1.00 plus the start fee.

### 🔗 Related Actors

More from the NexGenData **Social, video & reviews** family:

- [Google Play Store Scraper — App Metadata & ASO Data API](https://apify.com/nexgendata/google-play-store-scraper?fpr=2ayu9b)
- [iOS App Store Reviews Scraper — No API Key, Pay Per Record](https://apify.com/nexgendata/ios-app-store-reviews-scraper?fpr=2ayu9b)
- [Podcast Episodes Scraper — Episode Metadata API](https://apify.com/nexgendata/podcast-episodes-scraper?fpr=2ayu9b)
- [Review Intelligence to Notion — Cross-Platform Review Feed API](https://apify.com/nexgendata/review-intelligence-notion?fpr=2ayu9b)
- [StackOverflow Scraper — Q\&A & Dev Trends](https://apify.com/nexgendata/stackoverflow-questions?fpr=2ayu9b)
- [Universal Video Extractor — Any URL to Downloadable Video](https://apify.com/nexgendata/universal-video-extractor?fpr=2ayu9b)

6 more in this family on the [NexGenData Store page](https://apify.com/nexgendata?fpr=2ayu9b).

### 📊 What you get

Clean JSON output (6 fields):

- `url` — Url
- `title` — Title
- `description` — Description
- `markdown_content` — Markdown content
- `content_length` — Content length
- `scraped_at` — Scraped at

**Pricing:** $0.010 per page (Pay-Per-Event) — about **100 pages per $1**.

### 🤖 Use with AI agents

Point Claude, the OpenAI Agents SDK, an n8n flow or any MCP-aware client at it.

**Sample agent prompt:**

```
Search the web for 'what is the Apify Store' and return the top pages as clean Markdown for an LLM.
```

**Agentic payments (x402):** Supports agentic payment via x402 — call this actor with USDC, no API key required.

### 🔗 Related tools — Web crawling for AI/RAG

[Website Content Crawler](https://apify.com/nexgendata/website-content-crawler)

***

*A NexGenData utility actor.*

# Actor input Schema

## `query` (type: `string`):

A search query (the actor searches and reads top results) or a direct URL to read.

## `maxResults` (type: `integer`):

Maximum pages to fetch and return. Defaults to 5 (evaluation-sized); raise it for full runs (each result is billed).

## `maxContentLength` (type: `integer`):

Maximum characters of cleaned content to return per page.

## `includeMetadata` (type: `boolean`):

If enabled, includes page metadata (title, description, etc.).

## Actor input object example

```json
{
  "query": "best practices for RAG pipelines 2026",
  "maxResults": 3,
  "maxContentLength": 10000,
  "includeMetadata": true
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset of RAG Web Browser — Web Content for AI & LLMs records (6 fields per row: url, title, description, markdown_content, content_length, scraped_at).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "best practices for RAG pipelines 2026",
    "maxResults": 3,
    "maxContentLength": 10000,
    "includeMetadata": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("nexgendata/rag-web-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "best practices for RAG pipelines 2026",
    "maxResults": 3,
    "maxContentLength": 10000,
    "includeMetadata": True,
}

# Run the Actor and wait for it to finish
run = client.actor("nexgendata/rag-web-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "best practices for RAG pipelines 2026",
  "maxResults": 3,
  "maxContentLength": 10000,
  "includeMetadata": true
}' |
apify call nexgendata/rag-web-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nexgendata/rag-web-browser"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pSCboSU6lzfrCbBrd/builds/ZXp1T5QxrMzpwcpJa/openapi.json