# Substack Scraper: Posts, Comments & Notes (`sourabhbgp/substack-scraper`) Actor

Scrape any Substack newsletter, post or author profile: full article text, engagement counts, nested comment threads, reactors and Substack Notes. Search a newsletter, filter by date, free or paid. Custom domains work. No login. From $0.30 per 1,000 posts.

- **URL**: https://apify.com/sourabhbgp/substack-scraper.md
- **Developed by:** [Sourabh Kumar](https://apify.com/sourabhbgp) (community)
- **Categories:** News, Lead generation, Social media
- **Features:** Limited permissions
- **Stats:** 145 total users, 52 monthly users, 96.7% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Substack scraper: posts, full articles, comments, reactors and Notes

Get Substack data in three ways: every post of a newsletter, single posts by link, or an author's Substack Notes. Paste links or handles, press Start, and download posts with full text, engagement and comment threads.

**$0.30 per 1,000 posts or Notes. No login, cookies or proxy needed.**

Works with custom domains and `.substack.com` addresses alike, in any language.

<br>

### Pick what to paste

Fill one section, or both. Every input works alone or mixed with the others.

| You want | Paste into |
|---|---|
| Every post of a newsletter, with filters and search | `urls` (newsletter links) |
| One post, with its comments | `urls` (post or share links) |
| The newsletter of an author | `urls` (`substack.com/@handle` links) |
| An author's Substack Notes | `notesHandles` (handles) |

Every input can be scheduled. Run it weekly and you have a newsletter or Notes tracker.

<br>

### Why this Substack scraper?

- 📚 **The whole archive, not just the latest 20.** A 1,500-post newsletter comes back complete in under two minutes.
- 🎯 **Filters that fill your limit.** Ask for 40 paid posts from 2026 and you get 40.
- 🔍 **Search inside a newsletter**, with the full text of every match.
- 💬 **Full comment threads**, nested replies included, ready to rebuild as a tree.
- 🔗 **Any Substack link works:** a newsletter, a post, a share link from the app, or a profile.
- 🧾 **Never billed twice.** The same post reached through two links is one row, and posts your filters drop are free.

<br>

### 1. Newsletters: every post, with filters and search

**Paste** newsletter links, for example `https://newsletter.pragmaticengineer.com`, or a profile link like `https://substack.com/@lenny`.

**Get** one row per post: title, full text, author and co-authors, publish date, reactions, comment and restack counts, free or paid, tags, audio, and the newsletter's name.

| Option | Field | Default | What it does |
|---|---|---|---|
| Max posts per newsletter | `maxPosts` | 50 | `0` for the whole archive |
| Sort | `sortBy` | `newest` | `newest` or `oldest` |
| Add full article text | `includeContent` | `true` | Turn off for faster metadata-only runs |
| Content format | `contentFormat` | `both` | `html`, `text` or `both` |

**Filters** (all optional)

| Filter | Field |
|---|---|
| Only posts whose title or text match a keyword | `searchKeyword` |
| Free or paid posts only | `audienceFilter` |
| Newsletter, podcast, thread or video posts only | `typeFilter` |
| Published on or after / on or before a date (`YYYY-MM-DD`) | `dateFrom`, `dateTo` |

**Input**

```json
{
  "urls": ["https://newsletter.pragmaticengineer.com"],
  "audienceFilter": "paid",
  "dateFrom": "2026-01-01",
  "maxPosts": 50
}
```

**Output** (one real row, selected fields)

```json
{
  "id": 221215977,
  "title": "Why Global Trepanation?",
  "slug": "why-global-trepanation",
  "url": "https://www.astralcodexten.com/p/why-global-trepanation",
  "author": "Scott Alexander",
  "authorHandle": "astralcodexten",
  "publishedAt": "2026-10-21T21:53:11.440Z",
  "updatedAt": "2026-10-22T00:54:19.173Z",
  "contentText": "A friend asked me to investigate the following mystery: why do 5-10% of skulls from 10,000 BC to 3,0...",
  "wordCount": 1307,
  "readingTimeMinutes": 6,
  "coverImageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/73a7ca5d-8a51-4c67-9dd2-9348f3d6ffc3_2340x1316.jpeg",
  "audienceType": "everyone",
  "isPaywalled": false,
  "reactionCount": 383,
  "reactions": { "❤": 383 },
  "commentCount": 204,
  "restackCount": 18,
  "type": "newsletter",
  "hasAudio": true,
  "newsletter": {
    "name": "Astral Codex Ten",
    "description": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "url": "https://www.astralcodexten.com"
  },
  "comments": null,
  "facepile": null
}
```

<br>

### 2. Single posts: one post by link, with its comments

**Paste** post links (`https://www.astralcodexten.com/p/why-global-trepanation`) or share links from the Substack app. Filters and the post limit do not apply to a post you name.

**Get** the same row as above for each post. Turn on the extras to add the comment thread and the readers who reacted.

| Option | Field | Default | What it does |
|---|---|---|---|
| Add comments | `includeComments` | `false` | The full thread, nested replies included. Charged per comment |
| Add reactors and restackers | `includeFacepile` | `false` | Up to 5 of each, as Substack shows them. Charged per entry |

The extras work on newsletter links too: every post gets its own thread.

**Input**

```json
{
  "urls": ["https://www.astralcodexten.com/p/why-global-trepanation"],
  "includeComments": true,
  "includeFacepile": true
}
```

**Output** (one real comment from `comments`; `parentId` and `depth` rebuild the thread)

```json
{
  "id": 366989167,
  "parentId": 366969428,
  "depth": 1,
  "bodyText": "I wish we had a like button.",
  "authorName": "Greg Gentschev",
  "authorHandle": "gentschev",
  "publishedAt": "2026-10-21T22:31:56.846Z",
  "isPinned": false,
  "isDeleted": false,
  "reactionCount": 5,
  "reactions": { "❤": 5 }
}
```

<br>

### 3. Notes: an author's Substack Notes

**Paste** Substack handles, with or without `@`, for example `sahilbloom`.

**Get** one row per note, newest first: the text, reactions, restacks, reply count, and the author's bio and newsletter.

| Option | Field | Default | What it does |
|---|---|---|---|
| Max Notes per handle | `maxNotesPerHandle` | 50 | Up to 20,000 |
| Published on or after / before | `dateFrom`, `dateTo` | empty | Only Notes in this date range |

**Input**

```json
{
  "notesHandles": ["sahilbloom", "tedgioia"],
  "maxNotesPerHandle": 500,
  "dateFrom": "2026-01-01"
}
```

**Output** (one real row)

```json
{
  "id": 368220860,
  "handle": "sahilbloom",
  "authorName": "Sahil Bloom",
  "authorBio": "NYT Bestselling Author of The 5 Types of Wealth. Gave up a grand slam on ESPN in 2012 and still wait...",
  "publishedAt": "2026-10-23T12:28:21.063Z",
  "bodyText": "I believe that 99% of success is just the ability to outlast uncertainty. The one who can tolerate t...",
  "reactionCount": 217,
  "reactions": { "❤": 217 },
  "restackCount": 13,
  "childrenCount": 38,
  "publicationName": "The Curiosity Chronicle",
  "publicationUrl": "https://sahilbloom.substack.com"
}
```

<br>

### Use it from an AI agent

Connect Claude, Cursor or any MCP client to Apify's MCP server with this scraper as a tool:

```
https://mcp.apify.com?tools=sourabhbgp/substack-scraper
```

Then ask in plain words. The agent fills in the input, and you pay the same per result. For example:

- "Get the 20 newest posts from newsletter.pragmaticengineer.com with their full text."
- "Find every Lenny's Newsletter post about pricing and summarise them."
- "Pull all comments on this Substack post and tell me what readers disagree with."
- "Export Sahil Bloom's Substack Notes since January."

<br>

### What people use it for

- 📊 **Content research:** every post a newsletter has published, with engagement, to see what lands.
- 🧠 **AI and text analysis:** clean full text for summaries, topic models or a searchable archive.
- 💬 **Audience research:** read what readers actually say in the comments.
- 🎯 **Lead generation:** find active commenters and reactors around a topic, with their own newsletters.
- 🏁 **Competitor tracking:** watch rival newsletters' publishing pace, paid share and podcast output.

<br>

### How much does it cost?

The same price on every Apify plan. Full article text costs nothing extra.

| What you get | Per 1,000 |
|---|---|
| Posts or Notes | $0.30 |
| Comments (`includeComments`) | $0.30 |
| Reactors and restackers (`includeFacepile`) | $0.20 |

Each run also costs $0.00005 to start.

- **Free plan:** your $5 monthly credit covers about **16,600 posts**.
- **Starter plan ($29):** about **96,600 posts** a month.

<br>

### Limitations

- **Paid posts come back as the free preview.** Substack shows the full text only to paying subscribers.
- **Paid posts have no comments.** Substack shows them only to subscribers, so `comments` is empty even when `commentCount` is above zero.
- **Up to 5 reactors and 5 restackers per post,** whatever the reaction count. Their handle and bio stay empty.
- **Video posts usually come back as `podcast`.** That's how Substack labels them, so `typeFilter: "video"` rarely matches.
- **`authorTwitter` is usually empty.** Substack no longer sends it.

<br>

### FAQ

#### How do I scrape Substack without coding?

Paste a newsletter or post link, or a handle under Notes, and press **Start**. Download the results as Excel, CSV or JSON.

#### Can I get a newsletter's whole archive?

Yes. Set **Max posts per newsletter** to `0`. Set **Sort** to `oldest` to start from the first post.

#### Why did I get fewer posts than I asked for?

The newsletter has fewer posts that match your filters. The run's status message says so, and you pay only for posts delivered.

#### Do I need a Substack account or a proxy?

No. It reads only what any logged-out visitor can see, and handles blocks by itself.

#### Can I use it from code or an AI agent?

Yes. Call it through the Apify API, or connect it to an AI agent through Apify's MCP server.

#### Is scraping Substack legal?

It collects only what Substack shows any visitor, with no login and no paywall bypass. How you use the data is your responsibility, especially personal data such as commenter names.

#### Your feedback

Something missing or wrong? [Open an issue](https://console.apify.com/actors/YtOb83sq5rupKexLA/issues).

# Actor input Schema

## `urls` (type: `array`):

Substack newsletter URLs (URL mode). A post link (/p/...), a share link (open.substack.com) or a profile link (substack.com/@handle) also works. Provide either urls or notesHandles.

## `maxPosts` (type: `integer`):

Maximum number of posts to scrape per newsletter. Set to 0 for all posts.

## `searchKeyword` (type: `string`):

If set, scrape only posts whose title or body matches this keyword. Uses Substack's archive search endpoint.

## `audienceFilter` (type: `string`):

Filter posts by audience: all, free (audience=everyone), or paid (audience=only_paid).

## `typeFilter` (type: `string`):

Filter by post type.

## `dateFrom` (type: `string`):

Only include posts and Notes published on or after this date (YYYY-MM-DD).

## `dateTo` (type: `string`):

Only include posts and Notes published on or before this date (YYYY-MM-DD).

## `sortBy` (type: `string`):

Sort posts by date.

## `includeContent` (type: `boolean`):

Extract full article HTML and plain text. Disable for faster metadata-only scraping.

## `contentFormat` (type: `string`):

Format for article content output.

## `includeComments` (type: `boolean`):

Fetch the full nested comment thread for each post. Adds one HTTP call per post and is billed at $0.0003 per comment. Substack shows comments on paid posts only to subscribers, so paid posts return no comments.

## `includeFacepile` (type: `boolean`):

Fetch the users Substack lists as having reacted to and restacked each post (Substack shows up to 5 of each). One HTTP call per post; billed at $0.0002 per reactor/restacker.

## `notesHandles` (type: `array`):

Substack handles to scrape Notes for (Notes mode). Provide either urls or notesHandles.

## `maxNotesPerHandle` (type: `integer`):

Max Notes to return per handle.

## `proxyConfiguration` (type: `object`):

Optional. Blocked requests are already retried on a new IP automatically, so most runs do not need a proxy. Set one if you want every request to go through your own proxy. Apify Proxy bandwidth is billed separately by Apify.

## Actor input object example

```json
{
  "urls": [
    "https://newsletter.pragmaticengineer.com"
  ],
  "maxPosts": 50,
  "audienceFilter": "all",
  "typeFilter": "all",
  "sortBy": "newest",
  "includeContent": true,
  "contentFormat": "both",
  "includeComments": false,
  "includeFacepile": false,
  "maxNotesPerHandle": 50
}
```

# Actor output Schema

## `posts` (type: `string`):

Posts extracted from Substack newsletters including title, content, author, engagement metrics, and more.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://newsletter.pragmaticengineer.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sourabhbgp/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://newsletter.pragmaticengineer.com"] }

# Run the Actor and wait for it to finish
run = client.actor("sourabhbgp/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://newsletter.pragmaticengineer.com"
  ]
}' |
apify call sourabhbgp/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sourabhbgp/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YtOb83sq5rupKexLA/builds/JIq9XtTzPY06oHWW3/openapi.json