# Substack Scraper — Posts, Comments & Newsletter Data (`automation-lab/substack-scraper`) Actor

Scrape Substack newsletters and custom domains: full posts as HTML and plain text, comments, subscriber counts, and topic search. Full archive depth. For research, lead lists and AI pipelines. Export JSON, CSV, Excel, or use the API.

- **URL**: https://apify.com/automation-lab/substack-scraper.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** Social media, Lead generation, News
- **Features:** Limited permissions
- **Stats:** 806 total users, 174 monthly users, 98.4% runs succeeded, 11 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 post scraped (metadata)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper

### Next step: publication and partnership prospecting

Use Substack Scraper for newsletter/publication research, partnership prospecting, sponsorship research, and advertising outreach planning — not spam harvesting. When a public publisher domain is present, use [Website Contact Finder](https://apify.com/automation-lab/website-contact-finder) or [Social Media Profile Finder](https://apify.com/automation-lab/social-media-profile-finder) to find public contact channels, then verify any email addresses with [Bulk Email MX Verifier](https://apify.com/automation-lab/email-mx-verifier) or [Bulk SMTP Email Verifier](https://apify.com/automation-lab/smtp-email-verifier).

### Build a cold-outreach lead workflow on Apify

Use this actor as one step in a scrape → enrich → verify → export workflow:

1. **Source B2B accounts or contacts** with [LinkedIn Company Employees Scraper](https://apify.com/automation-lab/linkedin-company-employees-scraper), [LinkedIn Company Scraper](https://apify.com/automation-lab/linkedin-company-scraper), [Clutch.co Scraper](https://apify.com/automation-lab/clutch-scraper), [Yellow Pages Scraper](https://apify.com/automation-lab/yellowpages-scraper), [YellowPages Canada Business Scraper](https://apify.com/automation-lab/yellowpages-canada-scraper), or [Substack Scraper](https://apify.com/automation-lab/substack-scraper).
2. **Find public contact signals** with [Website Contact Finder](https://apify.com/automation-lab/website-contact-finder) or [Social Media Profile Finder](https://apify.com/automation-lab/social-media-profile-finder) when you have company domains/websites.
3. **Clean the list before outreach** with [Bulk Email MX Verifier](https://apify.com/automation-lab/email-mx-verifier) for fast syntax/MX/disposable/role-risk screening, then [Bulk SMTP Email Verifier](https://apify.com/automation-lab/smtp-email-verifier) when you need deeper mailbox and catch-all checks.
4. **Export to your outreach stack** as CSV/Excel/JSON for Instantly, HubSpot, Salesforce, Pipedrive, Airtable, Make, Zapier, or n8n.

Use only public/business contact data and comply with CAN-SPAM, GDPR, and your outreach platform's rules. Prefer company emails over personal addresses, filter role-based/risky emails when needed, and do not send to invalid or unverified addresses.

### What does Substack Scraper do?

**Substack Scraper** extracts data from any Substack newsletter -- posts with full HTML content, comments with nested replies, and publication metadata including subscriber counts. It supports **unlimited archive depth** (no 12-post cap), works with both `*.substack.com` and custom domain newsletters, and exports to JSON, CSV, Excel, or connects via API.

Unlike other scrapers, this actor uses Substack's public JSON API directly, with no browser. Temporary rate limits and dropped connections are retried automatically, and any input that still fails is listed in the run's status message.

Key capabilities:

- Discover public Substack posts and publications by topic keyword, even when you do not know their URLs
- Scrape full post archives from any Substack newsletter (no post limit)
- Extract complete HTML content, plain text ready for AI pipelines (`bodyText`), metadata, and engagement metrics for each post
- Fetch full comment threads with nested replies and author info
- Pull publication-level data: subscriber counts, description, language, logo, payments flag, author profiles
- Filter by date range, content type (newsletter/podcast/thread), or paywall status
- Automatically deduplicate overlapping URLs, newsletter archives, and keyword discoveries across a run
- Works with both `*.substack.com` and custom domain newsletters

### Who is it for?

- **Content strategists** analyzing newsletter trends, posting cadence, and topic coverage across the Substack ecosystem
- **Market researchers** benchmarking competitor newsletters by public subscriber count, engagement, and pricing models
- **Sales and marketing teams** building lead lists from newsletter author profiles and publication metadata
- **Data journalists** investigating media trends with structured datasets from newsletter archives
- **Academic researchers** studying online discourse through newsletter content and comment analysis
- **AI/ML engineers** building training datasets from high-quality long-form writing on Substack
- **Newsletter creators** auditing their own archive performance and comparing with peers

### Why use Substack Scraper?

- **Unlimited archive depth** -- Scrape the complete archive of any newsletter, including custom domains
- **Reliable public API access** -- Uses Substack's public JSON API, retries rate limits and dropped connections, and reports any input that still fails
- **Full comment threads** -- Extract comments with nested replies, reaction counts, and author metadata
- **Publication metadata** -- Subscriber counts, description, language, logo, payments flag and author info on every post
- **No proxy cost to you** -- Requests go directly to the API; a proxy is only used as a last retry, and pricing stays per result
- **Clean pay-per-event pricing** -- A small start fee and a price per result, with no completion charges. Runs that deliver nothing, such as bad input or an empty archive, are not charged
- **Every post as HTML and as plain text** -- Ready for spreadsheets, research, and AI/RAG pipelines without extra parsing
- **Custom domain support** -- Works with both `newsletter.substack.com` and custom domains like `www.lennysnewsletter.com`

Want Substack Notes instead of newsletter posts? Use [Substack Notes Search and Author Monitor](https://apify.com/automation-lab/substack-notes-search-author-monitor).

### What data can you extract from Substack?

**Per post (30+ fields):**

| Field | Description |
|-------|-------------|
| `title`, `subtitle`, `slug` | Post title, subtitle, and URL slug |
| `url` | Full canonical URL |
| `publishedAt`, `updatedAt` | Publication and update timestamps |
| `postType` | `newsletter`, `podcast`, or `thread` |
| `audience`, `isPaid` | Paywall status (`everyone` or `only_paid`) |
| `bodyHtml` | Full HTML content (free posts) |
| `bodyText` | The post as readable plain text, ready for AI, RAG and text analysis. `null` when content is off or the body is paywalled |
| `truncatedBodyText` | Preview text for all posts including paywalled |
| `wordcount` | Total word count (even for paid posts) |
| `coverImage` | Cover image URL |
| `description` | Post description/excerpt |
| `tags` | Post tags/categories |
| `reactionCount`, `commentCount`, `restacks` | Engagement metrics |
| `authorName`, `authorHandle`, `authorBio` | Author information |
| `authorPhotoUrl`, `authorId` | Author photo and ID |
| `publicationName`, `publicationId`, `publicationUrl` | Newsletter metadata |
| `publicationDescription`, `publicationLanguage`, `publicationLogoUrl`, `publicationPaymentsEnabled` | About text, language, logo and whether paid subscriptions are on (needs **Include Publication Info**) |
| `subscriberCount` | Rounded public subscriber count shown by Substack, such as `"1,200,000"` for 1.2M+. `null` when Substack shows no count |
| `podcastUrl`, `podcastDuration` | Podcast episode data (if applicable) |
| `hasVoiceover` | Whether the post has audio narration |
| `scrapedAt` | Timestamp when the data was collected |

**Per comment (12 fields):** `id`, `body`, `date`, `editedAt`, `name`, `handle`, `photoUrl`, `reactionCount`, `restacks`, `isAuthor`, `isPinned`, nested `replies`

**Per publication (14 fields):** `id`, `name`, `subdomain`, `customDomain`, `baseUrl`, `authorName`, `authorHandle`, `authorBio`, `authorPhotoUrl`, `subscriberCount`, `logoUrl`, `heroText`, `language`, `paymentsEnabled`

### How much does it cost to scrape Substack?

This actor uses **pay-per-event** pricing -- you pay only for what you scrape. No monthly subscription required. All platform compute costs are included in the per-event price.

| Event | FREE | BRONZE | SILVER | GOLD | PLATINUM | DIAMOND |
|---|---:|---:|---:|---:|---:|---:|
| **Start** | $0.005 | $0.005 | $0.005 | $0.005 | $0.005 | $0.005 |
| **Per post (metadata)** | $0.00115 | $0.001 | $0.00078 | $0.0006 | $0.0004 | $0.00028 |
| **Per post (with content)** | $0.0023 | $0.002 | $0.00156 | $0.0012 | $0.0008 | $0.00056 |
| **Per comment** | $0.000575 | $0.0005 | $0.00039 | $0.0003 | $0.0002 | $0.00014 |

From November 27, 2026 plan prices move to one standard scale. Starter (BRONZE) prices stay the same: posts with metadata $0.001 / $0.001 / $0.0009 / $0.0008 (FREE / BRONZE / SILVER / GOLD and up), posts with content $0.002 / $0.002 / $0.0018 / $0.0016, comments $0.0005 / $0.0005 / $0.00045 / $0.0004.

A run that delivers no posts is not charged, including the start fee.

**Real-world cost examples:**

| Scenario | Results | Duration | Cost (Free tier) |
|---|---|---|---|
| 1 newsletter, 50 posts (metadata) | 50 posts | ~3s | ~$0.06 |
| 1 newsletter, 50 posts (with content) | 50 posts | ~5s | ~$0.12 |
| 1 newsletter, 50 full-content posts + comments | 50 posts + ~200 comments | ~15s | ~$0.24 |
| 1 newsletter, full archive (500 full-content posts) | 500 posts | ~30s | ~$1.16 |
| 5 newsletters, 100 full-content posts each | 500 posts | ~60s | ~$1.16 |

These examples estimate FREE-tier event charges as one $0.005 start plus the listed result events; actual result counts determine the final charge.

Substack Scraper is one of the cheapest content scrapers on Apify. Because it uses direct API calls instead of a browser, there are no rendering costs and no proxy fees for you -- you only pay for the data you receive.

### How to scrape Substack newsletters step by step

1. Go to the [Substack Scraper](https://apify.com/automation-lab/substack-scraper) page on Apify Store
2. Click **Start for free** to open the actor configuration screen
3. Enter one or more newsletter URLs (e.g., `https://www.lennysnewsletter.com`), discovery keywords (e.g., `artificial intelligence`), or both
4. Choose your output options:
   - **Include Post Content** -- toggle on for full HTML body, off for metadata-only (faster and cheaper)
   - **Include Comments** -- toggle on to fetch comment threads with nested replies
   - **Include Publication Info** -- toggle on for subscriber counts and newsletter metadata
5. Optionally set filters:
   - **Content Type** -- limit to newsletters, podcasts, or threads
   - **Start/End Date** -- restrict to a date range (YYYY-MM-DD format)
   - **Free Posts Only** -- skip paywalled content
6. Set **Max Posts per Newsletter** (default 100, set to 0 for unlimited full archive)
7. Click **Start** and wait for results
8. Download your data in JSON, CSV, or Excel from the **Dataset** tab, or connect via API

### How do I scrape Substack newsletters and posts?

| Parameter | Type | Default | Description |
|---|---|---|---|
| `urls` | array | `[]` | Substack newsletter URLs. Accepts homepage, custom domain, post URLs, or /archive URLs |
| `keywords` | array | `[]` | Topics to discover across public Substack posts and publications |
| `maxSearchResultsPerKeyword` | integer | `20` | Matching posts to process per keyword (maximum `100`) |
| `maxPostsPerNewsletter` | integer | `100` | Max posts per newsletter. `0` = unlimited (full archive) |
| `includeContent` | boolean | `true` | Include full HTML body. Disable for metadata-only (faster, cheaper) |
| `includeComments` | boolean | `false` | Fetch comments for each post. Adds one API call per post |
| `includePublicationInfo` | boolean | `true` | Include newsletter metadata (subscriber count, pricing, author) |
| `contentType` | string | `all` | Filter: `all`, `newsletter`, `podcast`, or `thread` |
| `startDate` | string | -- | Only posts after this date (YYYY-MM-DD) |
| `endDate` | string | -- | Only posts before this date (YYYY-MM-DD) |
| `onlyFree` | boolean | `false` | Only include free posts. Skip paywalled content |

### What data can I extract from Substack?

```json
{
    "postId": 186226252,
    "title": "How to build AI product sense",
    "subtitle": "The secret is using Cursor for non-technical work",
    "slug": "how-to-build-ai-product-sense",
    "url": "https://www.lennysnewsletter.com/p/how-to-build-ai-product-sense",
    "publishedAt": "2026-02-03T13:45:58.303Z",
    "updatedAt": "2026-02-04T17:29:56.949Z",
    "postType": "newsletter",
    "audience": "everyone",
    "isPaid": false,
    "wordcount": 5867,
    "coverImage": "https://substackcdn.com/image/fetch/...",
    "tags": ["AI"],
    "reactionCount": 298,
    "commentCount": 31,
    "childCommentCount": 15,
    "restacks": 20,
    "hasVoiceover": false,
    "bodyHtml": "<div class=\"body markup\">...</div>",
    "authorName": "Tal Raviv",
    "authorHandle": "talsraviv",
    "publicationName": "Lenny's Newsletter",
    "subscriberCount": "1,100,000",
    "comments": [
        {
            "id": 209331673,
            "body": "This article creates a whole new paradigm for learning...",
            "date": "2026-02-03T15:34:25.318Z",
            "name": "Jack Cohen",
            "handle": "jackcohen10",
            "reactionCount": 9,
            "isAuthor": false,
            "replies": [
                {
                    "id": 209340123,
                    "body": "Thanks Jack!",
                    "name": "Tal Raviv",
                    "isAuthor": true,
                    "replies": []
                }
            ]
        }
    ],
    "scrapedAt": "2026-02-06T02:07:09.750Z"
}
```

### Tips for best results

- **Discover first by keyword** when you know a topic but not the relevant publication URLs; each match uses the normal post output fields
- **Start with metadata-only** (`includeContent: false`) to quickly survey a newsletter's archive before doing a full content scrape
- **Use date filters** to scrape only recent posts instead of full archives -- saves time and money
- **Comments are optional** -- each post with comments requires an extra API call, so only enable when needed
- **Paid posts** return all metadata (title, wordcount, reactions) but `bodyHtml` will be empty
- **Custom domains** work the same as `*.substack.com` URLs -- just paste the full URL
- **Use `maxPostsPerNewsletter: 0`** for unlimited archive depth -- scrapes every post ever published
- **Batch multiple newsletters** in a single run by adding multiple URLs -- more efficient than running separate jobs
- **Overlap is safe** -- when URLs or keyword results resolve to the same post, the first occurrence is output and charged only once per run
- **Filter by content type** to target only newsletter posts, podcast episodes, or threads separately

### Use with Claude AI (MCP)

This actor is available as a tool in Claude AI through the Model Context Protocol (MCP). Add it to Claude Desktop, Cursor, Windsurf, or any MCP-compatible client.

#### Setup for Claude Code

```bash
claude mcp add --transport http apify "https://mcp.apify.com"
```

#### Setup for Claude Desktop, Cursor, or VS Code

Add this to your MCP config file:

```json
{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=automation-lab/substack-scraper"
        }
    }
}
```

#### Example prompts

- "Scrape the last 30 posts from Lenny's Newsletter and summarize the main topics covered."
- "Get all free posts from this Substack newsletter published in 2025 with their engagement metrics."
- "Fetch the comments on this Substack post and tell me what readers think about the article."

Learn more in the [Apify MCP documentation](https://docs.apify.com/platform/integrations/mcp).

### Integrations

Connect Substack Scraper with your existing tools and automate newsletter data workflows:

- [Make](https://apify.com/integrations/make) -- Automate workflows triggered by new newsletter data
- [Zapier](https://apify.com/integrations/zapier) -- Connect to 5,000+ apps
- [Google Sheets](https://apify.com/integrations/google-sheets) -- Export directly to spreadsheets
- [Slack](https://apify.com/integrations/slack) -- Get notifications for new posts
- [GitHub](https://apify.com/integrations/github) -- Trigger workflows on new data
- [Webhooks](https://apify.com/integrations/webhooks) -- Send data to any endpoint

You can also schedule the scraper to run automatically at regular intervals. Set up a [schedule](https://docs.apify.com/platform/schedules) to monitor newsletters for new posts daily or weekly.

### API usage

**Node.js:**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('automation-lab/substack-scraper').call({
    urls: ['https://www.lennysnewsletter.com'],
    maxPostsPerNewsletter: 50,
    includeContent: true,
    includeComments: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**Python:**

```python
from apify_client import ApifyClient

client = ApifyClient('YOUR_API_TOKEN')

run = client.actor('automation-lab/substack-scraper').call(run_input={
    'urls': ['https://www.lennysnewsletter.com'],
    'maxPostsPerNewsletter': 50,
    'includeContent': True,
    'includeComments': False,
})

items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)
```

**cURL:**

```bash
curl "https://api.apify.com/v2/acts/automation-lab~substack-scraper/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -X POST -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.lennysnewsletter.com"], "maxPostsPerNewsletter": 50, "includeContent": true}'
```

#### RSS feed export for newsletter monitoring

For scheduled newsletter/post monitoring, create an Apify task and export the latest run as RSS with Apify's positional `outputFields` mapping:

```text
https://api.apify.com/v2/actor-tasks/{TASK_ID}/runs/last/dataset/items?format=rss&fields=title,url,description,publishedAt&outputFields=title,link,description,pubDate&token=YOUR_API_TOKEN
```

### Is it legal to scrape Substack?

Substack Scraper accesses only publicly available data through Substack's public API endpoints. It does not bypass any authentication, paywalls, or access controls. Paywalled content metadata (title, wordcount, engagement metrics) is returned, but the full body text of paid posts is not extracted.

Key points:

- The scraper uses the same public API that powers Substack's own website
- No login credentials are required or used
- Paywalled post content is not accessed or extracted
- The scraper respects rate limits with polite delays between requests
- Extracted data should be used in compliance with applicable laws and Substack's Terms of Service
- This Actor is not affiliated with, endorsed by, or sponsored by Substack Inc.

#### Failure diagnostics

Failed operations send sanitized diagnostic input, exceptions, and Actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL query strings are removed, and reports are retained for 30 days. Diagnostic credentials are not public input parameters.

Users are responsible for ensuring their specific use case complies with local data protection regulations (GDPR, CCPA, etc.) and Substack's Terms of Service. For more information, see the Apify blog on [web scraping legality](https://blog.apify.com/is-web-scraping-legal/).

### FAQ

#### How accurate is the subscriber count?

`subscriberCount` is the rounded figure Substack shows publicly, returned as a string. It is a lower bound: `1.2M+` becomes `"1,200,000"` and `9.8K+` becomes `"9,800"`. Counts under 1,000 are shown as is. Substack does not publish exact counts for larger newsletters, so small changes in a big newsletter won't appear between runs. The value is `null` when Substack shows no count for a publication, which is common when a publisher hides it.

**How fast is the scraper?**
Very fast. 50 posts (metadata only) complete in ~3 seconds. 50 posts with full content in ~5 seconds. Full archives of 500+ posts finish in under 30 seconds. No browser overhead; runs that hit a Substack rate limit can take a few seconds longer per retry.

**Can I scrape paid/paywalled posts?**
You get all metadata for paid posts (title, subtitle, wordcount, reactions, comments count) but `bodyHtml` will be empty since content access requires an active subscription.

**Does it work with custom domains?**
Yes. Enter the full URL (e.g., `https://www.lennysnewsletter.com`) and the scraper auto-detects it as a Substack newsletter.

**How many posts can I scrape?**
There is no limit. Set `maxPostsPerNewsletter: 0` to scrape the complete archive.

**Does it extract comments?**
Yes. Set `includeComments: true` to get full comment threads with nested replies, author info, and reaction counts. Each post with comments requires one extra API call.

**What about rate limits?**
Substack's public API has no detected rate limits. The scraper adds a polite delay between requests to be respectful.

**The `bodyHtml` field is empty for some posts.**
This is expected for paid/paywalled posts. The scraper returns all metadata (title, wordcount, reactions) but cannot access content behind a paywall without an active subscription.

**The scraper fails with "not a valid Substack" for a custom domain.**
Make sure you are using the full URL including `https://`. The scraper auto-detects Substack newsletters on custom domains, but the URL must be reachable and point to a valid Substack publication.

**Can I scrape multiple newsletters at once?**
Yes. Add multiple URLs to the `urls` input field. The scraper processes them sequentially and outputs all posts to a single dataset.

**What URL formats are accepted?**
The scraper accepts homepage URLs (`https://newsletter.substack.com`), custom domains (`https://www.lennysnewsletter.com`), individual post URLs, and `/archive` URLs. All formats are auto-detected.

### Related scrapers

- [Dev.to Scraper](https://apify.com/automation-lab/devto-scraper) -- Scrape articles and profiles from Dev.to
- [TechCrunch Scraper](https://apify.com/automation-lab/techcrunch-scraper) -- Extract articles and news from TechCrunch
- [Hacker News Scraper](https://apify.com/automation-lab/hackernews-scraper) -- Extract posts and comments from Hacker News
- [Reddit Scraper](https://apify.com/automation-lab/reddit-scraper) -- Scrape Reddit posts, comments, and subreddit data

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/substack-scraper/changelog.md

# Actor input Schema

## `urls` (type: `array`):

Substack newsletter URLs to scrape. Accepts homepage (e.g. https://newsletter.substack.com), custom domain (e.g. https://www.lennysnewsletter.com), post URLs, or /archive URLs.

## `keywords` (type: `array`):

Optional topics to search across public Substack posts, for example artificial intelligence or climate policy. Keyword results use the same post output as URL scraping.

## `maxSearchResultsPerKeyword` (type: `integer`):

Maximum matching posts to process for each discovery keyword.

## `maxPostsPerNewsletter` (type: `integer`):

Maximum number of matching posts to deliver per newsletter, after filters. Set to 0 for unlimited (full archive).

## `includeContent` (type: `boolean`):

Include the full body of each post as HTML (bodyHtml) and plain text (bodyText). Disable for metadata-only scraping (faster, cheaper).

## `includeComments` (type: `boolean`):

Fetch comments for each post. Adds one extra API request per post.

## `maxCommentsPerPost` (type: `integer`):

Maximum number of comments to return per post (including replies). Set to 0 for unlimited. Only applies when 'Include Comments' is enabled.

## `includePublicationInfo` (type: `boolean`):

Include newsletter metadata (subscriber count, description, language, logo, payments flag, author info). One extra request per newsletter.

## `contentType` (type: `string`):

Filter by post type.

## `startDate` (type: `string`):

Only include posts published after this date (YYYY-MM-DD format).

## `endDate` (type: `string`):

Only include posts published before this date (YYYY-MM-DD format).

## `onlyFree` (type: `boolean`):

Only include free posts. Skip paywalled content.

## Actor input object example

```json
{
  "urls": [
    "https://www.lennysnewsletter.com"
  ],
  "keywords": [
    "artificial intelligence"
  ],
  "maxSearchResultsPerKeyword": 10,
  "maxPostsPerNewsletter": 10,
  "includeContent": true,
  "includeComments": false,
  "maxCommentsPerPost": 20,
  "includePublicationInfo": true,
  "contentType": "all",
  "startDate": "2024-01-01",
  "endDate": "2026-12-31",
  "onlyFree": false
}
```

# Actor output Schema

## `overview` (type: `string`):

Dataset containing all scraped or keyword-discovered Substack posts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.lennysnewsletter.com"
    ],
    "keywords": [
        "artificial intelligence"
    ],
    "maxSearchResultsPerKeyword": 10,
    "maxPostsPerNewsletter": 10,
    "maxCommentsPerPost": 20,
    "startDate": "2024-01-01",
    "endDate": "2026-12-31"
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://www.lennysnewsletter.com"],
    "keywords": ["artificial intelligence"],
    "maxSearchResultsPerKeyword": 10,
    "maxPostsPerNewsletter": 10,
    "maxCommentsPerPost": 20,
    "startDate": "2024-01-01",
    "endDate": "2026-12-31",
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.lennysnewsletter.com"
  ],
  "keywords": [
    "artificial intelligence"
  ],
  "maxSearchResultsPerKeyword": 10,
  "maxPostsPerNewsletter": 10,
  "maxCommentsPerPost": 20,
  "startDate": "2024-01-01",
  "endDate": "2026-12-31"
}' |
apify call automation-lab/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/PqIVgDh92wdcdn9fj/builds/efx4M5EhRLRNRnQU9/openapi.json