# Substack Newsletter Scraper - Articles, Metadata & Full Content (`hata1234/substack-scraper`) Actor

Extract articles, metadata, and content from any Substack newsletter via public API. No proxy needed. Supports multiple newsletters, full article body extraction, audience filtering (free/paid), date range, keyword search, and pagination. Works with both substack.com subdomains and custom domains.

- **URL**: https://apify.com/hata1234/substack-scraper.md
- **Developed by:** [Moris Chao](https://apify.com/hata1234) (community)
- **Categories:** News, Social media
- **Features:** Limited permissions
- **Stats:** 35 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Newsletter Scraper

Scrape articles from any Substack newsletter using the public Substack API. Supports both `subdomain.substack.com` URLs and custom domains (e.g., `www.lennysnewsletter.com`).

### Features

- Scrape article metadata (title, author, date, reactions, comments, etc.)
- Optionally fetch full article HTML body
- Filter by audience (free/paid), content type, date range, or keyword
- Scrape multiple newsletters in a single run (comma-separated URLs)
- Concurrent batch fetching for full article bodies
- Polite rate limiting (500ms delay between requests)

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `newsletterUrl` | string | *required* | Newsletter URL(s). Comma-separated for multiple. |
| `maxItems` | integer | `50` | Max articles per newsletter. `0` = unlimited. |
| `sortBy` | string | `"new"` | `"new"` (newest first) or `"top"` (most popular). |
| `includeBody` | boolean | `false` | Fetch full HTML body for each article. |
| `audienceFilter` | string | `"all"` | `"all"`, `"free"`, or `"paid"`. |
| `typeFilter` | string | `"all"` | `"all"`, `"newsletter"`, `"podcast"`, or `"thread"`. |
| `dateFrom` | string | — | Only articles on/after this date (YYYY-MM-DD). |
| `dateTo` | string | — | Only articles on/before this date (YYYY-MM-DD). |
| `searchKeyword` | string | — | Filter by keyword in title or description. |

#### Example Input

```json
{
    "newsletterUrl": "https://www.lennysnewsletter.com",
    "maxItems": 10,
    "sortBy": "new",
    "includeBody": false
}
```

#### Multiple Newsletters

```json
{
    "newsletterUrl": "https://www.lennysnewsletter.com, https://stratechery.com",
    "maxItems": 20
}
```

### Output

Each article is saved to the default dataset with the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `id` | number | Substack post ID |
| `title` | string | Article title |
| `subtitle` | string | Article subtitle |
| `slug` | string | URL slug |
| `url` | string | Full canonical URL |
| `postDate` | string | ISO 8601 publish date |
| `audience` | string | `"everyone"` or `"only_paid"` |
| `type` | string | `"newsletter"`, `"podcast"`, or `"thread"` |
| `wordcount` | number | Word count |
| `reactions` | object | Reaction counts (e.g., `{"❤": 532}`) |
| `commentCount` | number | Number of comments |
| `coverImage` | string | Cover image URL |
| `author` | string | Author name(s) |
| `description` | string | Article description/excerpt |
| `body` | string | Full HTML body (only when `includeBody: true`) |

#### Example Output

```json
{
    "id": 123456,
    "title": "How to build a great product",
    "subtitle": "Lessons from top PMs",
    "slug": "how-to-build-a-great-product",
    "url": "https://www.lennysnewsletter.com/p/how-to-build-a-great-product",
    "postDate": "2026-03-03T13:45:17.054Z",
    "audience": "everyone",
    "type": "newsletter",
    "wordcount": 2642,
    "reactions": { "❤": 532 },
    "commentCount": 9,
    "coverImage": "https://substackcdn.com/image/...",
    "author": "Lenny Rachitsky",
    "description": "A deep dive into product excellence..."
}
```

### Notes

- **Paid articles**: Articles marked as `"only_paid"` may only return a preview of the body content.
- **Rate limiting**: The Actor adds a 500ms delay between API requests to avoid overloading Substack servers.
- **Custom domains**: Both `newsletter.substack.com` and custom domains like `platformer.news` are supported.
- **No authentication required**: Uses Substack's public API endpoints.

### Cost

This Actor uses only HTTP API calls (no browser), so it's very lightweight:

- \~256MB memory is sufficient
- No proxy required
- A typical run of 50 articles completes in under a minute

# Actor input Schema

## `newsletterUrl` (type: `string`):

URL of the Substack newsletter to scrape. Supports both subdomain.substack.com and custom domains (e.g. www.lennysnewsletter.com). Use comma-separated URLs to scrape multiple newsletters.

## `maxItems` (type: `integer`):

Maximum number of articles to scrape per newsletter. Set to 0 for unlimited.

## `sortBy` (type: `string`):

Sort order for articles.

## `includeBody` (type: `boolean`):

Whether to fetch the full HTML body for each article. This requires an additional API call per article and will slow down the scrape.

## `audienceFilter` (type: `string`):

Filter articles by audience type.

## `typeFilter` (type: `string`):

Filter articles by content type.

## `dateFrom` (type: `string`):

Only include articles published on or after this date (YYYY-MM-DD format).

## `dateTo` (type: `string`):

Only include articles published on or before this date (YYYY-MM-DD format).

## `searchKeyword` (type: `string`):

Filter articles whose title or description contains this keyword (case-insensitive).

## Actor input object example

```json
{
  "newsletterUrl": "https://www.lennysnewsletter.com",
  "maxItems": 50,
  "sortBy": "new",
  "includeBody": false,
  "audienceFilter": "all",
  "typeFilter": "all"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "newsletterUrl": "https://www.lennysnewsletter.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("hata1234/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "newsletterUrl": "https://www.lennysnewsletter.com" }

# Run the Actor and wait for it to finish
run = client.actor("hata1234/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "newsletterUrl": "https://www.lennysnewsletter.com"
}' |
apify call hata1234/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hata1234/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/q1qjHwY8rpJWw2dLV/builds/OnnoefeFzh0RluOAm/openapi.json