# Substack Scraper - Posts, Newsletters & Comments (`scraper_guru/substack-scraper`) Actor

Substack scraper for posts, newsletters, authors, subscriber data, full public content, engagement, and nested comments. Search by keyword or scrape any publication.

- **URL**: https://apify.com/scraper_guru/substack-scraper.md
- **Developed by:** [LIAICHI MUSTAPHA](https://apify.com/scraper_guru) (community)
- **Categories:** Social media, News, Lead generation
- **Features:** Limited permissions
- **Stats:** 77 total users, 10 monthly users, 100.0% runs succeeded, 5 bookmarks
- **User rating**: 4.54 out of 5 stars

## Pricing

from $2.40 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper - Posts, Newsletters & Comments

Scrape **Substack posts, newsletters, authors, publication details, subscriber data, engagement metrics, public article content, and nested comments**. Start from publication URLs, custom domains, direct post URLs, or search keywords.

The Actor uses lightweight public web endpoints instead of a browser, so routine runs are fast and use only 256 MB of memory. Results are available as JSON, CSV, Excel, XML, HTML, and RSS through the Apify Dataset API.

### Features

- Discover Substack publications by keyword, topic, author, or niche
- Scrape complete publication archives or individual `/p/` URLs
- Extract headlines, descriptions, dates, authors, tags, images, language, and word counts
- Collect reactions, comments, replies, restacks, and free or paid access status
- Return public article content as plain text, Markdown, and HTML
- Preserve nested public comment threads with author and reaction details
- Include publication name, description, logos, payment status, and visible subscriber counts
- Filter by date, post type, access, reactions, comments, and word count
- Support standard `substack.com` publications and Substack custom domains
- Produce a `STATS` record with per-publication results and run duration

### Use Cases

- **Competitor monitoring:** track publishing frequency, topics, authors, and engagement
- **Newsletter discovery:** find publications in a market before building a prospect list
- **Content research:** compare headlines, tags, formats, and audience response
- **AI and RAG datasets:** export clean Markdown or text with source metadata
- **Media monitoring:** schedule recurring snapshots of selected newsletters
- **Audience research:** analyze public conversations in comments and replies
- **Lead generation:** identify relevant publications, authors, and custom domains by niche

### Input

Provide at least one publication URL or search keyword.

| Field | Type | Default | Description |
|---|---|---:|---|
| `substackUrls` | array | optional | Publication URLs, custom domains, or direct post URLs |
| `searchQueries` | array | optional | Keywords used to discover relevant publications |
| `maxSearchResultsPerQuery` | integer | `10` | Publications discovered per keyword |
| `maxPostsPerSubstack` | integer | `50` | Matching posts per publication; `0` means the full archive |
| `includeContent` | boolean | `true` | Return public HTML, Markdown, and plain text |
| `includeComments` | boolean | `false` | Return public comments and nested replies |
| `maxCommentsPerPost` | integer | `20` | Maximum comments and replies per post |
| `includePublicationInfo` | boolean | `true` | Add publication and visible subscriber metadata |
| `postType` | string | `all` | `all`, `newsletter`, `podcast`, or `thread` |
| `startDate` | date | optional | Include posts published on or after this date |
| `endDate` | date | optional | Include posts published on or before this date |
| `onlyFree` | boolean | `false` | Exclude posts marked for paid subscribers |
| `minReactions` | integer | `0` | Minimum reaction count |
| `minComments` | integer | `0` | Minimum comment count |
| `minWordCount` | integer | `0` | Minimum word count |
| `postConcurrency` | integer | `10` | Parallel post requests, from 1 to 25 |

#### Scrape publications and comments

```json
{
  "substackUrls": [
    "https://www.lennysnewsletter.com",
    "https://platformer.news"
  ],
  "maxPostsPerSubstack": 20,
  "includeContent": true,
  "includeComments": true,
  "maxCommentsPerPost": 30,
  "startDate": "2026-01-01",
  "onlyFree": true,
  "postConcurrency": 10
}
```

#### Discover newsletters by keyword

```json
{
  "searchQueries": ["artificial intelligence", "product management"],
  "maxSearchResultsPerQuery": 5,
  "maxPostsPerSubstack": 10,
  "includeContent": false,
  "minReactions": 10
}
```

### Output

Each Dataset item represents one post. Existing fields such as `headline`, `date`, `likes`, `comments`, and `article_text` remain available, with richer fields added.

```json
{
  "record_type": "post",
  "post_id": 208730073,
  "publication_id": 10845,
  "substack_url": "https://www.lennysnewsletter.com",
  "post_url": "https://www.lennysnewsletter.com/p/example-post",
  "slug": "example-post",
  "headline": "Example newsletter post",
  "subheading": "A public subtitle",
  "post_type": "newsletter",
  "published_at": "2026-08-18T15:40:06.921Z",
  "date": "Aug 18, 2026",
  "free_or_paid": "Free",
  "is_paywalled": false,
  "likes": 346,
  "comments": 17,
  "child_comments": 14,
  "restacks": 9,
  "word_count": 933,
  "language": "en",
  "tags": ["Career"],
  "cover_image_url": "https://...",
  "author_name": "Example Author",
  "author_handle": "example",
  "author_url": "https://substack.com/@example",
  "article_text": "Plain text content...",
  "article_markdown": "## Markdown content...",
  "article_html": "<div>HTML content...</div>",
  "content_type": "full",
  "comment_threads": [
    {
      "id": 123,
      "body": "Useful analysis.",
      "author_name": "Reader",
      "reaction_count": 4,
      "is_author": false,
      "replies": []
    }
  ],
  "comments_collected": 1,
  "publication": {
    "name": "Example Newsletter",
    "url": "https://example.substack.com",
    "hero_text": "Publication description",
    "subscriber_count": 12000,
    "subscriber_count_label": "Over 12,000 subscribers",
    "payments_enabled": true,
    "logo_url": "https://..."
  },
  "search_query": "product management",
  "scraped_at": "2026-08-23T10:00:00Z"
}
```

`content_type` is `full`, `preview_only`, `metadata_only`, or `failed`. The Actor never bypasses a paywall; paid posts contain only content Substack exposes publicly.

### How to Use

1. Open the Actor and select **Try for free**.
2. Add publication or post URLs, search keywords, or both.
3. Set the number of posts and optional filters.
4. Enable comment extraction only when comment bodies are needed.
5. Start the run and open the **Output** tab.
6. Export the overview or content view in the format required by your workflow.

For a fast competitor snapshot, disable `includeContent`. For AI or RAG exports, keep it enabled and use `article_markdown` or `article_text`.

### API Examples

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scraper_guru/substack-scraper").call(run_input={
    "substackUrls": ["https://www.lennysnewsletter.com"],
    "maxPostsPerSubstack": 25,
    "includeContent": True,
    "includeComments": True,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["headline"], item["post_url"])
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('scraper_guru/substack-scraper').call({
  searchQueries: ['AI newsletters'],
  maxSearchResultsPerQuery: 5,
  maxPostsPerSubstack: 10,
  includeContent: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

### Pricing

This Actor uses pay-per-event pricing, so platform usage is included. You pay only for posts that are saved to the Dataset.

| Plan | Per 1,000 posts (from November 9, 2026) | Per 1,000 posts (until November 8, 2026) |
|---|---:|---:|
| Free | `$3.00` | `$0.50` |
| Bronze | `$2.80` | `$0.45` |
| Silver | `$2.60` | `$0.40` |
| Gold, Platinum, Diamond | `$2.40` | `$0.35` |

Each run also has a start charge of `$0.00035`. Example: 20 posts from each of 5 newsletters (100 posts) costs about `$0.30` on the Free plan. Apify shows the exact plan-adjusted price before each run.

### Reliability and Limits

- The Actor collects public data and does not log in or bypass paid-content restrictions.
- Visible subscriber counts are returned when Substack exposes them on the publication page.
- Keyword search discovers publications through Substack profiles, then applies your post settings to each publication.
- Very large full-archive runs take longer and produce more results. Test with 5 to 20 posts first.
- Comment extraction adds one request per post and can increase run duration.
- Substack can change its public web endpoints. Report reproducible extraction problems in the Actor's **Issues** tab.

### FAQ

#### Can it scrape a Substack custom domain?

Yes. Use the custom publication URL, such as `https://platformer.news`, or a standard `https://name.substack.com` URL.

#### Can it scrape one specific post?

Yes. Add a direct URL containing `/p/`. The Actor processes that post instead of the full publication archive.

#### Can it extract full paid posts?

No. It returns only the preview and metadata available to the public. `content_type` is set to `preview_only` for paid posts.

#### How does keyword discovery work?

The Actor searches public Substack profiles and publications matching each keyword, deduplicates the publications, and then scrapes their newest matching posts.

#### How do I get only recent, popular posts?

Set `startDate`, `minReactions`, and `minComments`. Filters are applied before full content and comments are requested, reducing unnecessary work.

#### Why is a subscriber count missing?

Some publications do not display a public count. The field remains empty instead of estimating a value.

#### Which content format should I use for AI pipelines?

Use `article_markdown` when headings and links matter, or `article_text` for embeddings and simple text analysis.

#### Does it support scheduled monitoring?

Yes. Create an Apify Task from a working input and schedule it. Connect the Dataset to a webhook, Make, n8n, Zapier, Google Sheets, or your own API.

### Related scrapers

Building a newsletter or audience research dataset? These Actors from the same developer pair well with this one:

| Actor | What it collects |
|---|---|
| [Beehiiv Newsletter Scraper](https://apify.com/scraper_guru/beehiiv-scraper) | Posts, authors and publication details from any Beehiiv newsletter |
| [Reddit Comments Deep Scraper](https://apify.com/scraper_guru/reddit-comments-deep-scraper) | Full Reddit comment threads with nested replies, for audience research |
| [YouTube Comments Extractor](https://apify.com/scraper_guru/youtube-comments-extractor) | Comments and replies from YouTube videos |

# Actor input Schema

## `substackUrls` (type: `array`):

Substack publication URLs, custom domains, or direct /p/ post URLs.

## `searchQueries` (type: `array`):

Find relevant Substack publications by topic, author, or niche, then scrape their posts.

## `maxSearchResultsPerQuery` (type: `integer`):

Maximum publications discovered for each search keyword.

## `maxPostsPerSubstack` (type: `integer`):

Maximum matching posts saved from each publication. Set to 0 for the full archive.

## `includeContent` (type: `boolean`):

Return available HTML, Markdown, and plain text. Disable for faster metadata-only runs.

## `includeComments` (type: `boolean`):

Return public comments and nested replies for each post.

## `maxCommentsPerPost` (type: `integer`):

Maximum comments and replies collected per post when comments are enabled.

## `includePublicationInfo` (type: `boolean`):

Add publication name, description, logos, subscriber count, payment status, and author details.

## `postType` (type: `string`):

Limit results to newsletters, podcasts, or threads.

## `startDate` (type: `string`):

Only include posts published on or after this date.

## `endDate` (type: `string`):

Only include posts published on or before this date.

## `onlyFree` (type: `boolean`):

Exclude posts marked for paid subscribers.

## `minReactions` (type: `integer`):

Only include posts with at least this many reactions.

## `minComments` (type: `integer`):

Only include posts with at least this many comments.

## `minWordCount` (type: `integer`):

Only include posts with at least this many words.

## `postConcurrency` (type: `integer`):

Number of post requests processed in parallel.

## Actor input object example

```json
{
  "substackUrls": [
    "https://www.lennysnewsletter.com"
  ],
  "maxSearchResultsPerQuery": 10,
  "maxPostsPerSubstack": 5,
  "includeContent": true,
  "includeComments": false,
  "maxCommentsPerPost": 20,
  "includePublicationInfo": true,
  "postType": "all",
  "onlyFree": false,
  "minReactions": 0,
  "minComments": 0,
  "minWordCount": 0,
  "postConcurrency": 10
}
```

# Actor output Schema

## `posts` (type: `string`):

Structured post records with metadata, content, publication details, and optional comment threads.

## `contentExport` (type: `string`):

Plain text, Markdown, HTML, and comments for content pipelines.

## `runStats` (type: `string`):

Discovered publications, saved posts, failures, settings, and duration.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "substackUrls": [
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerSubstack": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("scraper_guru/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "substackUrls": ["https://www.lennysnewsletter.com"],
    "maxPostsPerSubstack": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("scraper_guru/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "substackUrls": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerSubstack": 5
}' |
apify call scraper_guru/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scraper_guru/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CkRdoOVHA3D5zhDkn/builds/TCrjrm5bu3DS4J5CG/openapi.json