# Discourse Forum Scraper – Topics, Posts, Replies & Categories (`rl1987/discourse-community-scraper`) Actor

Scrape any Discourse forum — community.openai.com, community.n8n.io, discuss.pytorch.org, community.home-assistant.io and thousands more. Get rows for categories, topics and posts: title, URL, author, views, likes, tags, accepted-answer flag, and post bodies as HTML, Markdown and plain text.

- **URL**: https://apify.com/rl1987/discourse-community-scraper.md
- **Developed by:** [R.L.](https://apify.com/rl1987) (community)
- **Categories:** Social media
- **Features:** Limited permissions
- **Stats:** 9 total users, 4 monthly users, 96.9% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 dataset items

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Discourse Forum Scraper — Topics, Posts, Replies & Categories from Any Community

### What does this Discourse forum scraper do?

**Discourse Forum Scraper** extracts categories, topics and posts from any **Discourse-powered community forum** — no per-site configuration needed. Discourse powers most modern developer and product communities, and instead of parsing HTML this actor talks to the built-in JSON API every Discourse site exposes (`/latest.json`, `/c/{category}.json`, `/search.json`, `/t/{id}.json`, ...), which makes it fast, immune to theme and layout changes, and generic across sites. Point it at a forum (see the list below), choose what to crawl, and you get one row per category, topic and post.

Run it on the [Apify platform](https://apify.com) for scheduling, API access, webhooks, and proxy rotation without managing any infrastructure yourself.

### Why scrape a Discourse forum?

- **Community research & sentiment analysis** — pull recurring questions, complaints, and feature requests from a product's support forum.
- **Competitive intelligence** — monitor what users say about a competitor's product in its official community.
- **Support/knowledge-base mining** — build a searchable dataset of solved threads and accepted answers.
- **Content/dataset building** — collect real-world Q\&A pairs or discussion threads for training or analysis.

### Which Discourse forums does it work on?

Any of them. The actor checks `/site.json` — which every Discourse install serves — and refuses a host that isn't Discourse, so there is no per-site configuration and nothing to maintain when a forum changes its theme. These were each confirmed to answer with a valid Discourse `/site.json`:

| Forum | What it covers |
|---|---|
| [community.openai.com](https://community.openai.com) | OpenAI API and ChatGPT developer community |
| [community.n8n.io](https://community.n8n.io) | n8n automation: questions, jobs, showcases |
| [community.home-assistant.io](https://community.home-assistant.io) | Home Assistant smart-home support |
| [discuss.pytorch.org](https://discuss.pytorch.org) | PyTorch usage and troubleshooting |
| [discuss.streamlit.io](https://discuss.streamlit.io) | Streamlit apps and deployment |
| [discuss.elastic.co](https://discuss.elastic.co) | Elasticsearch, Kibana, Logstash |
| [forums.developer.nvidia.com](https://forums.developer.nvidia.com) | CUDA, Jetson, drivers |
| [forum.arduino.cc](https://forum.arduino.cc) | Arduino hardware and sketches |
| [forums.docker.com](https://forums.docker.com) | Docker Desktop and Engine |
| [community.grafana.com](https://community.grafana.com) | Grafana dashboards and data sources |
| [users.rust-lang.org](https://users.rust-lang.org) | Rust language help |
| [forum.obsidian.md](https://forum.obsidian.md) | Obsidian plugins and workflows |
| [forum.rclone.org](https://forum.rclone.org) | rclone configuration and bug reports |
| [community.smartthings.com](https://community.smartthings.com) | SmartThings devices and automations |
| [discourse.ubuntu.com](https://discourse.ubuntu.com) | Ubuntu announcements and docs discussion |
| [community.wanikani.com](https://community.wanikani.com) | WaniKani Japanese-learning community |
| [meta.discourse.org](https://meta.discourse.org) | Discourse's own forum about Discourse |

Self-hosted and private-company Discourse instances work the same way, as long as the content you point at is public.

### How do I run it?

1. Set **Forum URL** to the base URL of any Discourse forum (e.g. `https://community.n8n.io`).
2. Pick a **Mode**:
   - `categories` — list every category and **subcategory** on the forum, so you can find the right `categorySlug` without guessing (run this first if you don't already know it).
   - `latest` / `top` — the site-wide topic list.
   - `category` — one category (set **Category slug/path** to a category item's `category_slug_for_scraping` from a `categories` run, e.g. `questions/12`).
   - `search` — Discourse's search endpoint (set **Search query**; supports Discourse search filters like `order:latest #category`).
   - `topicUrls` — scrape only the specific topic URLs you provide.
3. Toggle **Include full posts** to also fetch every reply in each topic (off = topic metadata only, much faster).
4. Set **Max topics** / **Max posts per topic** to bound the run, and hit **Start**.

### What inputs does it take?

See the **Input** tab for the full schema. Key fields:

| Field | Description |
|---|---|
| `forumUrl` | Base URL of the Discourse forum |
| `mode` | `categories`, `latest`, `top`, `category`, `search`, or `topicUrls` |
| `categorySlug` | Category slug/path (mode=`category`) |
| `searchQuery` | Search query (mode=`search`) |
| `topicUrls` | List of direct topic URLs (mode=`topicUrls`). Through the API or MCP, pass objects: `[{"url": "https://forum.example/t/slug/123"}]` |
| `includePosts` | Fetch full post stream per topic |
| `maxTopics` / `maxPostsPerTopic` | Limits (0 = unlimited) |
| `requestDelaySecs` / `maxConcurrency` | Politeness/rate-limit controls |
| `proxyConfiguration` | Apify Proxy settings |

### What does a result row look like?

Up to three item shapes land in the same dataset, distinguished by `type`:

```json
{
  "type": "category",
  "forum": "https://community.n8n.io",
  "category_id": 13,
  "title": "Jobs",
  "slug": "jobs",
  "url": "https://community.n8n.io/c/jobs/13",
  "parent_category_id": null,
  "topic_count": 615,
  "category_slug_for_scraping": "jobs/13"
}
```

```json
{
  "type": "topic",
  "forum": "https://community.n8n.io",
  "topic_id": 302663,
  "title": "AI Assistant on self-hosted n8n: early setup instructions",
  "slug": "ai-assistant-on-self-hosted-n8n-early-setup-instructions",
  "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663",
  "posts_count": 4,
  "views": 512,
  "like_count": 3,
  "created_at": "2026-01-05T10:00:00.000Z"
}
```

```json
{
  "type": "post",
  "topic_id": 302663,
  "post_id": 565721,
  "post_number": 1,
  "username": "Ophir_Prusak",
  "cooked_html": "<p>Here's how to set it up:</p><ol><li>Install the <strong>self-hosted</strong> package</li><li>Restart n8n</li></ol>",
  "content_markdown": "Here's how to set it up:\n\n1. Install the **self-hosted** package\n2. Restart n8n",
  "content_text": "Here's how to set it up: Install the self-hosted package Restart n8n",
  "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663/1"
}
```

`cooked_html` is Discourse's raw rendered post body. `content_markdown` and `content_text` are cleaned conversions of it (via [markdownify](https://pypi.org/project/markdownify/) and [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)) — use these for LLM input, search indexing, or anywhere raw HTML is unwanted.

Download the dataset as JSON, CSV, Excel, or HTML from the **Storage** tab.

#### Which fields does each item type contain?

**`type: "category"`** — one row per category, from `mode: categories`:

| Field | Description |
|---|---|
| `forum` | Base URL of the forum the row came from |
| `category_id` | Discourse's numeric category ID |
| `title` | Category name |
| `slug` | Category slug as it appears in the URL |
| `url` | Link to the category |
| `parent_category_id` | Parent category's ID, or `null` for a top-level category |
| `posts_count` | Posts in the category, as Discourse reports it |
| `topic_count` | Topics in the category |
| `category_slug_for_scraping` | `slug/id` string to paste straight into `categorySlug` |

**`type: "topic"`** — one row per topic:

| Field | Description |
|---|---|
| `forum`, `topic_id`, `slug`, `url` | Where the topic lives |
| `title` | Topic title |
| `category_id` | Category the topic sits in |
| `tags` | Array of the topic's tag names, as strings (Discourse returns these as objects on some forums and strings on others; both come out as names here) |
| `created_at` / `last_posted_at` | ISO timestamps of the first and latest post |
| `posts_count` | Posts in the topic, including the first one |
| `reply_count` | Replies as Discourse counts them |
| `views` | View count. Discourse's search endpoint does not return it, so in `mode: search` it is `null` unless **Include full posts** is on |
| `like_count` | Likes on the topic. Same caveat as `views` in `mode: search` |
| `pinned`, `closed`, `archived` | Topic state flags |
| `has_accepted_answer` | `true` when a reply is marked as the solution |

**`type: "post"`** — one row per post, when **Include full posts** is on:

| Field | Description |
|---|---|
| `forum`, `topic_id`, `topic_title` | Thread the post belongs to |
| `post_id`, `post_number`, `url` | The post's own ID, its position in the thread, and a direct link |
| `username` / `name` | Author's forum handle and display name |
| `created_at` / `updated_at` | ISO timestamps |
| `reply_to_post_number` | Post number this one replies to, or `null` for a top-level reply |
| `reply_count` | Replies to this post |
| `like_count` | Likes on the post |
| `cooked_html` | Rendered HTML of the post body |
| `content_markdown` | Post body as clean Markdown |
| `content_text` | Post body as clean plain text |

#### What do the Discourse terms mean?

- **cooked** — Discourse's own name for a post body after its Markdown has been rendered to HTML. `cooked_html` is that HTML verbatim; `content_markdown` and `content_text` are conversions of it, so quotes, code blocks and lists survive as text.
- **post stream** — the ordered list of post IDs in a topic. The actor walks it in chunks of 20 so a 500-reply thread is fetched completely, not just the first page.
- **post_number** — a post's 1-based position in its topic. `1` is the original post; everything above it is a reply.
- **accepted answer** — a reply marked as the solution on forums that run Discourse's Solved plugin. `has_accepted_answer` is `true` on those topics and absent or `null` on forums without the plugin.
- **slug** — the human-readable part of a Discourse URL. `category_slug_for_scraping` pairs it with the numeric ID (`jobs/13`), which is the form `categorySlug` wants.

### How do I turn a forum into research? (worked example)

You can run this actor end-to-end through the [Apify MCP server](https://mcp.apify.com/) to answer research questions without writing scraping code. Example: *"What kind of freelance work do n8n users want to buy in 2026?"*

1. **Discover the category** — call the actor with `mode: "categories"` on `https://community.n8n.io`. This lists every category and subcategory with a ready-to-use `category_slug_for_scraping`, e.g. `"jobs/13"` for the `Jobs` category.
2. **Scrape it with posts** — call the actor again with `mode: "category"`, `categorySlug: "jobs/13"`, `includePosts: true`, `maxTopics: 80`. This returns each job-post topic plus its replies, with `content_text` already cleaned of HTML.
3. **Analyze** — pull `content_text` across all `post` items and count keyword mentions (e.g. "AI agent", "CRM", "voice AI", "RAG", "self-hosted") to see what buyers are actually asking for, and compare "hiring"/"looking for" vs. "for hire"/"available" topic titles to gauge demand vs. supply.

A real run against `community.n8n.io/c/jobs` (61 topics, 204 rows) found: **AI agent building** was the top request (72 mentions), usually combined with **CRM integration** (92), **document/RAG/OCR processing** (60/38/22), and a growing **voice AI** niche (Vapi + Twilio + n8n stacks, 31+23 mentions). Buy-side posts outnumbered for-hire posts roughly 2:1, and **self-hosted** n8n (21 mentions) was a common hard requirement. Rates ranged $15–50/hr or $250–1,500 flat, with a shift toward long-term retainers over one-off builds.

This same discover → scrape → analyze flow works for any Discourse forum's category (support backlogs, feature requests, showcase/built-with sections, etc.), not just job boards.

### How much does it cost?

The current price of every event is on the **Pricing** tab of this Actor's page in Apify Store. That tab is generated from the Actor's own pricing record, so it is always current, and your plan's rung is applied automatically. A figure repeated here would be a second copy that drifts out of date.

You can cap what any single run may spend with **Max total charge**.

### Tips for a clean crawl

- Leave `includePosts` off for a quick topic-index crawl, then re-run with `topicUrls` on just the topics you care about.
- Lower `maxConcurrency` / raise `requestDelaySecs` on smaller or self-hosted forums to stay polite.
- Use `mode: search` with Discourse's own filters (`order:latest`, `#category`, `after:2026-01-01`) when you want a slice of a big forum rather than its whole topic list.

### FAQ

**Do I need an account, an API key or a login?**

No. Every Discourse forum serves a JSON twin of each page (`/latest.json`, `/c/{slug}.json`, `/search.json`, `/t/{slug}/{id}.json`) and the actor reads those anonymously. The flip side: categories that require a login are invisible to it, because they are invisible to an anonymous visitor.

**How do I find the right category slug?**

Run `mode: categories` once. Each category row carries `category_slug_for_scraping` (e.g. `jobs/13`) — paste that into `categorySlug` with `mode: category`. No guessing from URLs.

Subcategories are included, and most forums keep most of their categories in them: `forum.arduino.cc` has 7 top-level categories and 70 subcategories, `discourse.ubuntu.com` 10 and 100. A subcategory's `parent_category_id` tells you which category it sits under.

**Does it get every reply in a long thread?**

Yes. It reads the topic's full post stream and then fetches the posts that weren't inlined, 20 IDs per request, so a 500-reply thread comes out complete. Set `maxPostsPerTopic` if you only want the first N posts of each thread in thread order.

**Can I scrape just a few specific threads?**

Yes — `mode: topicUrls` with the topic URLs. Each distinct host is validated once against `/site.json`, and a URL on a host that isn't Discourse is skipped with a warning instead of failing the run.

**Can I run it on a schedule?**

Yes. A scheduled `latest` or `search` run is a simple way to watch a community for new threads. The actor does not remember previous runs, so rows repeat across runs — de-duplicate on `topic_id` and `post_id` if you append them to your own store.

**Will it get me rate-limited or blocked?**

It is built to avoid that: `requestDelaySecs` (default 1 second) and `maxConcurrency` (default 3) throttle it, and an HTTP 429 triggers a back-off of 5, then 10, then 15 seconds before retrying. Lower the concurrency further on small or self-hosted forums, and enable Apify Proxy for large crawls.

**What happens if a page fails, or the site isn't Discourse?**

A host whose `/site.json` isn't Discourse ends the run immediately — finished, not failed — with a status message saying which of the three things happened: the site answered and is not Discourse, it refused the request (bot protection, so try a residential proxy), or it could not be reached. A run that could never have worked is not counted against the actor's failure rate. After that, failures are contained: a request is retried three times and then skipped with an error in the log, a 404 is logged and skipped, and a topic that throws is skipped with a warning while the rest of the crawl continues.

**Can I feed the posts straight into an LLM?**

That is what `content_markdown` and `content_text` are for — the same post body without the HTML, so a thread can go into a prompt, an embedding or a search index as-is. `cooked_html` stays in the row for when you need the original markup.

### Good to know

- Only scrape publicly accessible forum content, and respect each forum's Terms of Service and `robots.txt`.
- This actor is not affiliated with Discourse or with any of the forums named above.
- Found a Discourse site this doesn't work on? Open an issue in the Issues tab.

### Community research toolkit

Part of the **Community research toolkit** — research online communities and creators across forums, link-in-bio pages, and social platforms:

- [Reddit API Scraper](https://apify.com/rl1987/reddit-api-scraper) — Scrape Reddit posts, comments, search results, subreddits, and user profiles.
- [Linktree Profile Scraper](https://apify.com/rl1987/linktree-scraper) — Scrape profile info and all links from Linktree pages.
- [phpBB Forum Scraper](https://apify.com/rl1987/phpbb-forum-scraper) — Generic scraper for phpBB-powered forums.
- [Hacker News Scraper](https://apify.com/rl1987/hacker-news-scraper) — Scrape Hacker News by search, user, listing, or thread.
- [Social Blade Scraper — YouTube, TikTok, Instagram, Twitch Stats](https://apify.com/rl1987/social-blade-lookup) — Social Blade stats and growth projections for creators.

### Did you find this useful?

Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.

# Actor input Schema

## `forumUrl` (type: `string`):

Base URL of any Discourse-powered forum, e.g. https://community.n8n.io

## `mode` (type: `string`):

What to crawl. 'categories' lists the forum's categories/sections (use this to discover a categorySlug for mode=category). 'latest'/'top' pull the site-wide topic list, 'category' pulls one category, 'search' pulls search results, 'topicUrls' scrapes only the topics you list.

## `categorySlug` (type: `string`):

Required when mode=category. E.g. 'questions' or 'questions/12' (slug or slug/id) as it appears in the category URL.

## `searchQuery` (type: `string`):

Required when mode=search. Uses Discourse's /search.json endpoint (supports its query filters, e.g. 'order:latest #category').

## `topicUrls` (type: `array`):

Required when mode=topicUrls. Direct URLs to Discourse topics. Through the API or MCP, pass each one as an object: \[{"url": "https://forum.example/t/slug/123"}] - a bare list of strings is rejected before the run starts.

## `includePosts` (type: `boolean`):

If true, fetch every topic's full post stream (original post + all replies) in addition to the topic metadata. If false, only topic-level metadata is scraped (much faster, fewer requests). In Search mode this also fills `views` and `like_count`, which Discourse's search endpoint does not return.

## `maxTopics` (type: `integer`):

Maximum number of topics to scrape. 0 = unlimited.

## `maxPostsPerTopic` (type: `integer`):

Maximum number of posts to fetch per topic (only used when includePosts is true). 0 = unlimited.

## `requestDelaySecs` (type: `integer`):

Polite delay between outgoing API requests to avoid overloading the forum.

## `maxConcurrency` (type: `integer`):

Maximum number of concurrent HTTP requests.

## `proxyConfiguration` (type: `object`):

Proxy settings. Recommended if scraping many pages to avoid rate limiting.

## Actor input object example

```json
{
  "forumUrl": "https://community.n8n.io",
  "mode": "latest",
  "topicUrls": [],
  "includePosts": true,
  "maxTopics": 100,
  "maxPostsPerTopic": 0,
  "requestDelaySecs": 1,
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "forumUrl": "https://community.n8n.io"
};

// Run the Actor and wait for it to finish
const run = await client.actor("rl1987/discourse-community-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "forumUrl": "https://community.n8n.io" }

# Run the Actor and wait for it to finish
run = client.actor("rl1987/discourse-community-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "forumUrl": "https://community.n8n.io"
}' |
apify call rl1987/discourse-community-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rl1987/discourse-community-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/T7Yaojs7XA1pIPx5Y/builds/qVSQXUFP2X5Mw3OTY/openapi.json