# Substack Scraper - Find Newsletters, Posts & Comments (`webdata_labs/substack-intelligence-scraper`) Actor

\[$2.00 / 1K] Find Substack newsletters by category leaderboard or search, then scrape their posts, comments, authors and clean text. Ranked results, no URL list needed, no start fee.

- **URL**: https://apify.com/webdata_labs/substack-intelligence-scraper.md
- **Developed by:** [WebData Labs](https://apify.com/webdata_labs) (community)
- **Categories:** Social media, News, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 10.3% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 substack posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### 🔎 What does Substack Intelligence Scraper do?

**Substack Intelligence Scraper** finds Substack newsletters and turns them into structured post, comment, author, and publication datasets. Give it a category like Technology or Crypto and it returns the ranked leaderboard, then collects each newsletter's archive. You can also hand it `*.substack.com` publications, custom domains, archive pages, or individual post URLs.

Use it as a fast Substack API alternative for competitive research, newsletter monitoring, content analysis, sponsorship research, and AI-ready corpora. The Actor uses Substack's public HTTP endpoints and server-rendered data, so it does not need a browser or customer-managed proxy settings.

**You do not need to know the newsletters first.** Most Substack scrapers, including the largest one on the Store, require you to supply every publication URL yourself, and the leaderboard is sold as a separate Actor. This one does discovery and the archive in a single run.

Key capabilities include:

- Discovery by Substack category leaderboard, with the rank each newsletter holds
- Discovery by search phrase, for topics that do not map onto one category
- Three leaderboard rankings: all, paid bestsellers, and free
- Complete public archive pagination with `0` for unlimited depth
- Public post HTML, normalized plain text, and Markdown
- Nested comment threads plus optional analytics-friendly comment rows
- Publication, audience, author, tag, podcast, reaction, and restack metadata
- Date, post type, paywall, include-keyword, and exclude-keyword filters
- Direct collection from a single `/p/...` post URL
- Persistent `onlyNew` monitoring for scheduled runs
- Per-source and per-post error rows, so one bad URL does not discard a batch

### Ready-to-run examples

- [Top Technology Substack newsletters](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/top-technology-substack-newsletters) - ranked leaderboard, no URLs needed
- [Substack paid bestsellers by category](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/substack-paid-bestsellers-by-category) - who actually sells subscriptions
- [Top newsletters and their latest posts](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/top-newsletters-and-their-latest-posts) - discovery and archive in one run
- [Monitor new Substack newsletter posts](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/monitor-substack-newsletter-posts)
- [Export Substack comments and replies](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/export-substack-comments)
- [Build a Substack RAG dataset](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/build-substack-rag-dataset)

### 👥 Who is this Substack scraper for?

- Newsletter teams monitoring competitors and topic coverage
- Researchers comparing publishing cadence and reader engagement
- Sponsorship and partnership teams evaluating public publications
- Data journalists studying newsletter ecosystems
- AI teams preparing searchable text or RAG datasets
- Creators auditing their own public archives

### 🧭 Find newsletters without knowing their URL

Set **Substack categories** (or **Search phrases**) and the Actor returns a ranked list of
publications before it collects anything else. Each discovered row carries
`leaderboardRank`, `leaderboardCategory` and `leaderboardRanking`, so the ranking itself is
data you can sort and track, not just a way to find URLs.

- **33 categories**, from Technology and Finance to Crypto, Parenting and Games.
- **Three rankings.** `paid` is a genuinely different ordering of the same category, not a
  filter of `all`, so it answers "who actually sells subscriptions here?".
- **`discoveryOnly`** returns the ranked list and stops, which is much faster and cheaper
  when the list is all you need.
- Anything discovered flows into the normal pipeline, so every filter below (dates,
  keywords, paywall, post type) applies to newsletters you never had to name.

Discovered publications are then resolved through their homepage, which is where the
subscriber count comes from. The leaderboard payload itself does not carry one.

### ⚙️ How to scrape Substack newsletters

1. Add one or more newsletter homepages, custom domains, archive pages, or post URLs.
2. Keep the safe default of 50 posts per publication or use `0` for a complete archive.
3. Choose whether to fetch public content and select HTML, text, Markdown, or all formats.
4. Optionally enable comments and select nested, separate-row, or combined output.
5. Apply date, content-type, paywall, or keyword filters.
6. Start the Actor and open the Output tab.
7. Download JSON, CSV, Excel, XML, RSS, or connect through the Apify API.

Apify can run the same configuration on a schedule, send a webhook when it finishes, and connect the dataset to Make, Zapier, Google Sheets, cloud storage, or your own application.

### 📥 Input

```json
{
  "categories": ["Technology"],
  "leaderboardRanking": "paid",
  "maxPublicationsPerSource": 25,
  "discoveryOnly": false,
  "maxPostsPerNewsletter": 20
}
```

- `categories` - Substack categories to pull the ranked leaderboard for. Optional.
- `leaderboardRanking` - `all`, `paid`, or `free`. Default `all`.
- `searchQueries` - find publications by phrase instead of category. Optional.
- `maxPublicationsPerSource` - publications to take per category or phrase. Default `25`.
- `discoveryOnly` - return the ranked list and skip the archives. Default `false`.
- `urls` - explicit publication or post URLs. Optional once you use discovery.

Everything below still applies, and now applies to discovered newsletters too.

See the Input tab for every option. A typical run looks like this:

```json
{
    "urls": ["https://example.substack.com", "https://newsletter.example.com/p/a-public-post"],
    "maxPostsPerNewsletter": 100,
    "includeContent": true,
    "contentFormat": "all",
    "includeComments": true,
    "commentOutput": "both",
    "includePublicationInfo": true,
    "publicationOutput": "both",
    "contentType": "all",
    "onlyFree": false,
    "includeKeywords": ["AI", "product strategy"],
    "onlyNew": false
}
```

`maxPostsPerNewsletter` limits matching posts, not archive pages scanned. Keyword matching is case-insensitive and checks titles, subtitles, descriptions, public preview text, and tags.

### 📤 What Substack data can be extracted?

| Data              | Important fields                                                        |
| ----------------- | ----------------------------------------------------------------------- |
| Post identity     | `postId`, `title`, `subtitle`, `slug`, `url`                            |
| Timing and access | `publishedAt`, `updatedAt`, `postType`, `audience`, `isPaid`            |
| Public content    | `bodyHtml`, `bodyText`, `bodyMarkdown`, `truncatedBodyText`             |
| Engagement        | `reactionCount`, `reactions`, `commentCount`, `restacks`                |
| Discovery         | `tags`, `language`, `coverImage`, `description`, `wordCount`            |
| Authors           | `authors`, `authorName`, `authorHandle`, public bios and photos         |
| Publication       | name, URL, subscriber signals, branding, benefits, paid-plan status     |
| Comments          | body, date, author, reactions, pinned status, reply depth and parent ID |

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. The Output tab includes dedicated views for posts, comments, publications, and all records.

Example post row:

```json
{
    "itemType": "post",
    "postId": 123456789,
    "title": "A practical guide to product strategy",
    "url": "https://example.substack.com/p/product-strategy",
    "publishedAt": "2026-08-01T12:00:00.000Z",
    "postType": "newsletter",
    "isPaid": false,
    "wordCount": 2450,
    "tags": ["Product", "Strategy"],
    "reactionCount": 120,
    "commentCount": 18,
    "bodyText": "Public article content...",
    "bodyMarkdown": "## Public article content...",
    "publicationName": "Example Newsletter",
    "scrapedAt": "2026-08-09T12:00:00.000Z"
}
```

### 💵 How much does it cost to scrape Substack?

The recommended Store pricing is pay per event with no start fee: **$2.00 per 1,000 posts** and **$0.50 per 1,000 comments**, with lower prices for higher Apify subscription tiers. Publication and recoverable error rows are free. Comments are charged once when collected, whether you choose nested, separate, or combined comment output.

For cost-efficient runs:

- Start with `includeContent: false` to survey metadata quickly.
- Keep comments disabled unless reader discussion is required.
- Use a date window for recurring research.
- Keep the default 50-post cap for the first run.

### 🔁 How to monitor new Substack posts

Enable `onlyNew` and run the same input on an Apify schedule. The Actor stores emitted post IDs in a configuration-specific named key-value store. Later runs suppress previously delivered posts while preserving separate state for different URL and filter combinations.

For RSS delivery, request the last task dataset in RSS format and map `title`, `url`, `description`, and `publishedAt` to feed fields.

### 🛠️ Substack scraper API

Run the Actor from Node.js with `apify-client`:

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('webdata_labs/substack-intelligence-scraper').call({
    urls: ['https://on.substack.com'],
    maxPostsPerNewsletter: 20,
    contentFormat: 'text',
    onlyNew: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

The same input works through the REST API, Apify CLI, webhooks, schedules, and MCP integrations.

### ⚠️ Limits and responsible use

- **A category and ranking serves at most 525 publications.** Substack stops at page 20 and
  returns an error beyond it. Smaller categories run out well before that. The payload
  claims there is more even on the last page, so the number is the source's, not a setting.
- **Leaderboard rows carry no subscriber count.** Substack does not expose one on that
  endpoint. Subscriber counts come from each publication's own homepage, so they appear on
  publication and post rows rather than on discovery rows, and only when a publication
  publishes the figure.
- **Ranks are a snapshot.** The leaderboard is read at run time; there is no rank history.
  Schedule the Actor if you want a series.
- **Search finds fewer newsletters than a category.** Category leaderboards are the deeper
  surface; search is for topics that do not map onto one category.

The Actor collects only data available without authentication. It does not log in, bypass access controls, or return paid post bodies. Paid posts can still expose public metadata and preview text. Substack can change undocumented public endpoints or page payloads, so recoverable failures are emitted as structured `error` rows.

Public comments and author profiles can contain personal data. Use the dataset only for a legitimate purpose and comply with Substack's terms, GDPR, CCPA, copyright rules, and other applicable laws. If you are unsure whether your use is legitimate, consult qualified legal counsel.

### 🧩 Related Actors

- [Bluesky Scraper](https://apify.com/webdata_labs/bluesky-scraper) - posts, profiles and follower graphs on the open network.
- [Reddit Monitor & Scraper](https://apify.com/webdata_labs/reddit-signal-scraper) - brand mentions and buying signals in the threads people trust.
- [RedNote & Douyin Scraper](https://apify.com/webdata_labs/rednote-xiaohongshu-douyin-scraper) - what is trending on Xiaohongshu and Douyin before it reaches the West.
- [UFC Scraper](https://apify.com/webdata_labs/ufc-scraper) - fighters, fights, events and rankings from ufc.com.
- [Threads Scraper](https://apify.com/webdata_labs/threads-scraper) - posts, profiles and search on Meta's text network.

### ❓ FAQ and support

#### Does it support custom Substack domains?

Yes. Add the full custom-domain URL and the Actor discovers its canonical publication metadata.

#### Can it scrape a single post?

Yes. A URL containing `/p/post-slug` is treated as a direct post and does not expand into the full archive.

#### Why is a paid post body empty?

The Actor intentionally omits paid post bodies. Public metadata, engagement, tags, word count, and preview text can still be returned.

#### How are duplicate inputs handled?

Duplicate source strings, publication archives, and post IDs are deduplicated within each run. `onlyNew` extends post deduplication across scheduled runs with the same monitoring configuration.

For bugs or feature requests, use the Actor's Issues tab and include the public source URL plus the run ID. Do not include private credentials or subscription cookies.

# Actor input Schema

## `urls` (type: `array`):

Public Substack URLs to collect. Both \*.substack.com and custom publication domains are supported. Optional if you use Discovery below.

## `categories` (type: `array`):

Pull the ranked leaderboard for each selected Substack category. Substack serves at most 525 publications per category and ranking.

## `leaderboardRanking` (type: `string`):

Which leaderboard ordering to read. 'Paid' is a genuinely different ranking of the same category, not a filter of 'All'.

## `searchQueries` (type: `array`):

Find publications by keyword, for topics that do not map onto one category. Each phrase is searched separately.

## `maxPublicationsPerSource` (type: `integer`):

How many publications to take from each category or search phrase. The source ceiling is 525 per category and ranking.

## `discoveryOnly` (type: `boolean`):

Return the ranked publications and stop, without collecting each newsletter's posts. Much faster and cheaper when you only need the list.

## `maxPostsPerNewsletter` (type: `integer`):

Maximum matching posts returned for each publication. Use 0 to collect the complete public archive.

## `sortOrder` (type: `string`):

Collect the newest posts first or the publication's top posts first.

## `contentType` (type: `string`):

Return every post type or only newsletters, podcasts, or Substack threads.

## `startDate` (type: `string`):

Optional ISO date or timestamp. A YYYY-MM-DD value includes that entire UTC day.

## `endDate` (type: `string`):

Optional ISO date or timestamp. A YYYY-MM-DD value includes that entire UTC day.

## `onlyFree` (type: `boolean`):

Exclude subscriber-only and paid posts from the result dataset.

## `includeKeywords` (type: `array`):

Keep posts whose title, subtitle, description, preview, or tags contain at least one phrase. Matching is case-insensitive.

## `excludeKeywords` (type: `array`):

Drop posts whose title, subtitle, description, preview, or tags contain any listed phrase.

## `includeContent` (type: `boolean`):

Fetch public post pages to extract the body. Paid post bodies are never returned.

## `contentFormat` (type: `string`):

Choose HTML, clean text, Markdown, or multiple body representations.

## `includeComments` (type: `boolean`):

Fetch public comments, nested replies, public author handles, reactions, and pinned status for each selected post.

## `commentOutput` (type: `string`):

Nest comments inside each post, emit analytics-friendly comment rows, or return both formats.

## `includePublicationInfo` (type: `boolean`):

Attach public publication identity and audience metadata to results.

## `publicationOutput` (type: `string`):

Embed publication fields in posts, emit one publication row, or return both representations.

## `onlyNew` (type: `boolean`):

Remember emitted post IDs for this source and filter combination and suppress them in later runs.

## Actor input object example

```json
{
  "urls": [
    "https://on.substack.com"
  ],
  "categories": [],
  "leaderboardRanking": "all",
  "searchQueries": [],
  "maxPublicationsPerSource": 25,
  "discoveryOnly": false,
  "maxPostsPerNewsletter": 50,
  "sortOrder": "new",
  "contentType": "all",
  "onlyFree": false,
  "includeKeywords": [],
  "excludeKeywords": [],
  "includeContent": true,
  "contentFormat": "html_and_text",
  "includeComments": false,
  "commentOutput": "nested",
  "includePublicationInfo": true,
  "publicationOutput": "embedded",
  "onlyNew": false
}
```

# Actor output Schema

## `discovered` (type: `string`):

Ranked publications found by category leaderboard or search.

## `posts` (type: `string`):

Newsletter posts with engagement, authors, public content, and publication context.

## `comments` (type: `string`):

Flat public comment and reply records when separate comment output is enabled.

## `publications` (type: `string`):

Publication-level metadata when separate publication output is enabled.

## `allResults` (type: `string`):

Every post, comment, publication, and recoverable error row.

## `summary` (type: `string`):

Counts and monitoring status for the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://on.substack.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("webdata_labs/substack-intelligence-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://on.substack.com"] }

# Run the Actor and wait for it to finish
run = client.actor("webdata_labs/substack-intelligence-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://on.substack.com"
  ]
}' |
apify call webdata_labs/substack-intelligence-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,webdata_labs/substack-intelligence-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KQqfCNsPrPp1O3YXV/builds/bKppr5DcepHzx798l/openapi.json