# Discourse Scraper: Topics, Posts, Users & Search (`perconey/discourse-scraper`) Actor

Scrape any Discourse forum via the public REST API. Latest / top topics, category topics, full topic + posts, user profiles + activity, full-text search. No browser, no proxies, no auth. Pay only per result item.

- **URL**: https://apify.com/perconey/discourse-scraper.md
- **Developed by:** [Perconey](https://apify.com/perconey) (community)
- **Categories:** Developer tools, Social media, News
- **Features:** Limited permissions
- **Stats:** 7 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 result items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Discourse Scraper do?

**Discourse Scraper** pulls structured data from any [Discourse](https://www.discourse.org/) forum via the **official public REST API**. Topics with view counts and like counts, full post threads, user profiles with trust levels and badges, category trees, full-text search. The actor calls the documented public JSON endpoints directly: no browser, no proxies, no cookies, no auth. One actor works with **every Discourse-powered community**: HuggingFace, Django, Python.org, Unity, KiCad, Ruby on Rails, Brave, meta.discourse.org, and hundreds more.

Try it instantly: pick **getLatest**, leave instance as `https://discuss.huggingface.co`, click Start. You get the 30 newest HuggingFace forum topics in under 3 seconds for $0.03.

### Why use Discourse Scraper?

- **DevRel teams**: Monitor mentions of your project across the major open-source forums. Schedule daily `searchPosts` runs across Django, Python, HuggingFace, Unity in parallel.
- **Community managers**: Track engagement on your own Discourse forum. `getLatest` + `getCategoryTopics` give you topic counts, view counts, like counts for every recent thread.
- **Customer-support archaeology**: When a bug report references "the forum thread from last month", pull `getTopicDetail` with the topic id and you get the full conversation tree in JSON.
- **Recruiters**: `getUserProfile` returns trust level, badge count, post count - quick signals on technical depth in a community.
- **OSS maintainers**: Pull `getCategoryTopics` for "help" categories on multiple Discourse instances to see what users struggle with this week.

### How to use Discourse Scraper

1. Open the **Input** tab.
2. Pick an **action** from the dropdown. `getLatest` is the simplest starting point.
3. Set **instance** (default `https://discuss.huggingface.co`). To scrape a different Discourse forum, paste its URL.
4. For category / topic / user / search actions, fill **queries**.
5. Tune **maxItems** (default 30).
6. Click **Start**.

#### Query format by action

Action | Query format
\--- | ---
getLatest | leave empty
getTop | leave empty (use topPeriod field if needed)
getCategories | leave empty
getCategoryTopics | category slug (e.g. `beginners`) or `slug/id` (e.g. `beginners/5`)
getTopicDetail | numeric topic id (e.g. `175977`)
getUserProfile | username (e.g. `julien-c`)
getUserActivity | username
searchPosts | free-text search query

### Input

Field | Required | Description
\--- | --- | ---
`action` | yes | Which API call to make. Eight options.
`instance` | yes | Discourse forum URL. Default https://discuss.huggingface.co.
`queries` | sometimes | Required for category / topic / user / search actions.
`maxItems` | no | Max items per query. Default 30.
`topPeriod` | no | getTop only. all / yearly / quarterly / monthly / weekly / daily.

### Output

Every item carries `_type` (`topic` / `post` / `category` / `user` / `user_action` / `search_result` / `error`) plus `_action` and `_instance`.

```json
{
    "_type": "topic",
    "_action": "getLatest",
    "_instance": "https://discuss.huggingface.co",
    "id": 175977,
    "title": "Practical match for 128Gb Strix Halo with 2x3090s? (inference for coding)",
    "slug": "practical-match-for-128gb-strix-halo-with-2x3090s-inference-for-coding",
    "category_id": 5,
    "posts_count": 2,
    "views": 41,
    "like_count": 0,
    "created_at": "2026-05-14T10:08:00Z",
    "bumped_at": "2026-05-14T10:12:00Z",
    "tags": [],
    "url": "https://discuss.huggingface.co/t/practical-match-for-128gb-strix-halo-with-2x3090s-inference-for-coding/175977"
}
```

You can download the dataset in JSON, CSV, XML, Excel, RSS or HTML format from the Output tab.

### Data fields

Type | Key fields
\--- | ---
`topic` | id, title, slug, category_id, posts_count, views, like_count, created_at, bumped_at, last_posted_at, tags, archetype, closed, archived, pinned, url
`post` | id, topic_id, post_number, username, user_trust_level, cooked (HTML), raw (markdown), reply_count, like_count, accepted_answer, created_at, url
`category` | id, name, slug, description, topic_count, post_count, color, parent_category_id, url
`user` | id, username, name, title, trust_level, post_count, topic_count, badge_count, likes_given, likes_received, created_at, last_seen_at
`user_action` | action_type, action_code, created_at, excerpt, topic_id, topic_title, post_number, category_id, url
`search_result` | id, topic_id, post_number, title, blurb, username, like_count, url

### Pricing

**Pay-per-result: $0.001 per item.** No flat monthly fee.

Cost examples:

- Daily 30 newest HuggingFace topics: **$0.03**
- 1,000 topics from the HF "beginners" category: **$1.00**
- A 200-post thread with full posts: **$0.20**
- 50 user profiles across moderators of a forum: **$0.05**

### Tips

- **Discourse forums run different versions.** Most endpoints we wrap have been stable since 2018, but tag plugins are optional - we omit tag actions in v0.1 because they 404 on some installs.
- **Category slug auto-resolves.** Pass just `beginners` and the actor looks up the numeric id from `/categories.json` before fetching. You can also pass `beginners/5` if you already know it.
- **Topic detail returns chunks of 20 posts.** Past that, the actor fetches additional batches via `/t/{id}/posts.json?post_ids[]=...` until maxItems is reached.
- **Search is full-text.** It searches both posts and topics; the actor flattens results into a single `search_result` type with a `topic_id` so you can fetch the full thread separately.

### FAQ, disclaimers, support

**Is this legal?** The actor calls each Discourse forum's official public REST API with documented endpoints. Public read access is the design intent of the open-source Discourse software (GPL-licensed). We identify with a clear User-Agent and honor 429 / Retry-After.

**Does it work with private forums?** No. We only hit anonymous read endpoints. Forums that require login to view content are out of scope.

**Will I get rate-limited?** Discourse has generous per-IP rate limits for read traffic and the actor retries with exponential backoff on 429. For very heavy scraping consider supplying an API key via the headers in your own fork.

**Why are tags missing?** The tags plugin is optional and not enabled on every Discourse instance. The actor returns topic.tags when present but doesn't have a dedicated getTags action because the endpoint 404s too often.

**Bug or feature request?** Open an Issue on the actor's Issues tab. I usually respond within a day.

**Need a scraper for Hacker News, Stack Overflow, dev.to, arxiv, Lemmy, Mastodon, PeerTube?** See my other actors at https://apify.com/perconey.

# Actor input Schema

## `action` (type: `string`):

Pick the action. getLatest / getTop / getCategories work without queries. getCategoryTopics needs a category slug. getTopicDetail needs a numeric topic id. getUserProfile / getUserActivity need a username. searchPosts needs a query.

## `instance` (type: `string`):

Base URL of the Discourse forum. Examples: https://discuss.huggingface.co, https://forum.djangoproject.com, https://discuss.python.org, https://meta.discourse.org.

## `queries` (type: `array`):

One entry per query. getCategoryTopics: category slug (e.g. 'beginners') or 'slug/id' if known. getTopicDetail: numeric topic id (e.g. 175977). getUserProfile / getUserActivity: username (e.g. julien-c). searchPosts: free-text.

## `maxItems` (type: `integer`):

Stop after this many items. Latest / top / category feeds paginate 30 per page. Topic detail returns posts in chunks of 20.

## `topPeriod` (type: `string`):

Discourse 'top' period selector. Default is the forum's configured default (usually weekly or monthly).

## Actor input object example

```json
{
  "action": "getLatest",
  "instance": "https://discuss.huggingface.co",
  "queries": [],
  "maxItems": 30
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "instance": "https://discuss.huggingface.co",
    "queries": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("perconey/discourse-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "instance": "https://discuss.huggingface.co",
    "queries": [],
}

# Run the Actor and wait for it to finish
run = client.actor("perconey/discourse-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "instance": "https://discuss.huggingface.co",
  "queries": []
}' |
apify call perconey/discourse-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,perconey/discourse-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/goBIPyvB0eRz27wCc/builds/I0BlXuAc0mm4qqj1s/openapi.json