# Substack Posts Scraper (`fetch_cat/substack-posts-scraper`) Actor

Collect public Substack newsletter posts, archives, and metadata for content research and media monitoring.

- **URL**: https://apify.com/fetch_cat/substack-posts-scraper.md
- **Developed by:** [Fetch Cat](https://apify.com/fetch_cat) (community)
- **Categories:** Social media, News, Marketing
- **Features:** Limited permissions
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.02 / 1,000 post extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Export **public Substack posts** from publication URLs or custom domains into a structured dataset. Substack Posts Scraper collects titles, dates, public previews, authors and available engagement counts for newsletter research, content analysis and media monitoring.

It does not bypass paywalls. Body text and metadata depend on what each publication exposes publicly.

### Quick start

1. Open the Actor and enter a publication homepage or domain.
2. Start with five posts and keep the other settings unchanged.
3. Run the Actor, then open the dataset to review and export the results.

```json
{
  "publicationUrls": [{"url": "https://www.lennysnewsletter.com"}],
  "maxPostsPerPublication": 5,
  "includeBodyHtml": false,
  "includeRssFallback": true
}
```

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

https://youtu.be/1OW8gOqlZbY?t=25

### Ready-to-run examples

Choose an example to start with a small, preconfigured input.

- [Export Lenny’s Newsletter posts](https://apify.com/fetch_cat/substack-posts-scraper/examples/substack-lennys-newsletter-posts-export) — Collect up to ten public posts from one product newsletter.
- [Collect AI newsletter posts](https://apify.com/fetch_cat/substack-posts-scraper/examples/substack-ai-newsletter-posts) — Collect up to ten posts each from Import AI and One Useful Thing. These examples use the same extraction mode with different publication lists.

### What you get

- **Publication archives:** One dataset row per saved post, with public identifiers and source context.
- **Content and engagement:** Titles, previews, available body text, authors, reaction counts and comment counts. This is not comment-text extraction.
- **Focused collections:** Optional publication archive search and inclusive date filters.
- **Optional body HTML:** Include the public body HTML when available. Enabling this does not unlock subscriber-only content.

### Who is it for?

- **Researchers and journalists:** Compare public newsletter coverage and cite source post URLs.
- **Content teams:** Collect competitor topics and public engagement signals.
- **AI workflow builders:** Pass available post text to a separate summarization or classification step.

### Input settings

| Setting | JSON key | Meaning |
| --- | --- | --- |
| Publication URLs or domains | `publicationUrls` | Publication homepages or domains. Use URL objects as in Quick start. |
| Maximum posts per publication | `maxPostsPerPublication` | 1–1,000 saved posts per publication, default 20. |
| Search term | `searchTerm` | Optional publication archive search, not global Substack search. |
| Published after / before | `dateFrom`, `dateTo` | Inclusive dates in `YYYY-MM-DD` form. |
| Include body HTML | `includeBodyHtml` | Default false. Public text can still be returned without HTML. |
| Publication URLs to resume | `resumePublicationUrls` | Additional URLs processed before the main list. |
| Use feed fallback | `includeRssFallback` | Default true. A feed can return fewer fields or posts. |
| Publication concurrency | `concurrency` | 1–10 publications at a time, default 3. |

Provide at least one publication URL. The main and resume lists together support at most 100 entries. The form requires `publicationUrls`, so retain it when adding resume URLs.

For a topic-focused input, add `"searchTerm": "AI"` to the Quick start input. Search applies to publication archives only. Feed fallback does not apply the search term. Date filters apply to posts with a usable date, and records with missing or unparseable dates are retained.

### Output data

One row represents one saved post. Fields can be null when absent from the public response. Feed results usually expose fewer fields.

| Fields | Meaning |
| --- | --- |
| `publicationInput`, `publicationUrl`, `publicationDomain` | Original input and normalized publication context. |
| `postId`, `publicationId`, `canonicalUrl`, `slug` | Post and publication identifiers and public post location. |
| `title`, `subtitle`, `socialTitle`, `description` | Available titles and preview. |
| `postDate`, `updatedAt` | Publication and update dates as returned by the source. |
| `authorName`, `byline` | Public author attribution. |
| `audience`, `isPaidOnly`, `isLimited`, `type` | Available audience and content indicators. Feed flags do not establish paid-access status. |
| `bodyText`, `truncatedBodyText`, `bodyHtml` | Public text or preview. `bodyHtml` is included only when enabled. |
| `wordCount`, `coverImage` | Available length and image. |
| `reactionCount`, `commentCount` | Available counts, not reaction or comment records. |
| `tags`, `section`, `sourceEndpoint` | Tags, section and source context. |

Illustrative shape only, not a recorded extraction:

```json
{
  "publicationUrl": "https://example.substack.com",
  "canonicalUrl": "https://example.substack.com/p/example-post",
  "title": "Example post",
  "postDate": "2026-10-01T10:00:00.000Z",
  "authorName": null,
  "bodyText": "Public preview text",
  "reactionCount": null,
  "commentCount": null,
  "tags": []
}
```

The run’s key-value storage also contains `RUN_SUMMARY` with saved counts, publication failures and pending URLs when available. It is separate from the post dataset.

#### Download your results

Open your run's dataset, choose **Export** and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See [Apify's download and export guide](https://docs.apify.com/storage/dataset#apify-console) for the available formats and field options.

### Pricing

The `start` event is charged once after input validation when collection begins. A run can incur this charge even if no posts are saved.

The `post` event is charged for each public post saved to the dataset. Missing fields do not make a saved post free. Publication failures are summarized separately, not saved as charged post rows.

See the [live Pricing tab](https://apify.com/fetch_cat/substack-posts-scraper/pricing) for current rates and discounts.

### Tips for best results

- **Start small.** Try five posts from one publication before increasing your list.
- **Use publication homepages.** This Actor collects publication archives, not individual post URL lookups or global search results.
- **Check date coverage.** Narrow windows and unavailable older content can leave few or no matching posts.
- **Enable HTML only when needed.** It increases dataset size without making private content available.
- **Compare runs in your workflow.** A later run can return posts you have already collected.

### Limits and caveats

- Public data only. No login, subscriber-only bodies, email lists or paywall bypass.
- Search terms are not applied to feed fallback. Feeds may contain only recent entries.
- Results and field completeness depend on the publication. There is no guaranteed full archive export or selectable engagement sort.
- A run with no saved posts currently fails. Inspect `RUN_SUMMARY` for available failure or pending-work details rather than treating an empty run as proof that a publication is private or deleted.
- Some runs can return useful posts alongside publication failures. Review the summary before assuming every input completed.
- This is an independent tool, not an official Substack API.

### Automation and integrations

#### Save your settings for next time

Once you have an input that works for you, [save it as an Apify task](https://docs.apify.com/actors/running/tasks). This keeps your settings together so you can use them again without filling in the form from scratch.

#### Run on a schedule

Need fresh results regularly? You can [schedule the Actor or a saved task](https://docs.apify.com/actors/running/schedules) to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

#### Send results to your other tools

Use **n8n, Make or Zapier** to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the [Apify integration guides](https://docs.apify.com/integrations) for your chosen tool.

For a custom connection, an [Apify webhook](https://docs.apify.com/integrations/webhooks) can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

### API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its `defaultDatasetId` to fetch the results.

For setup instructions, see Apify's [JavaScript client guide](https://docs.apify.com/api/client/js), [Python client guide](https://docs.apify.com/api/client/python) and [Actor runs API reference](https://docs.apify.com/api/v2/acts-runs-post).

#### cURL

```bash
curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~substack-posts-scraper/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"publicationUrls":[{"url":"https://www.lennysnewsletter.com"}],"maxPostsPerPublication":5}'
```

#### Node.js

```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/substack-posts-scraper').call({
  publicationUrls: [{ url: 'https://www.lennysnewsletter.com' }],
  maxPostsPerPublication: 5,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("fetch_cat/substack-posts-scraper").call(run_input={
    "publicationUrls": [{"url": "https://www.lennysnewsletter.com"}],
    "maxPostsPerPublication": 5,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

### MCP and AI agents

You can ask your AI assistant to collect data with **Substack Posts Scraper** and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the [Apify MCP server](https://mcp.apify.com/). MCP gives your assistant a way to find this scraper, run it and read the results. Follow the [official setup and sign-in guide](https://docs.apify.com/mcp) for your app.

#### Easier setup with an Apify plugin

If you use a supported coding assistant, the [official Apify plugin](https://docs.apify.com/agents/plugins) can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

#### Use Apify AI in Console

Prefer to keep everything in Apify? Open [Apify AI in Console](https://docs.apify.com/account/apify-ai) and ask it to find Fetch Cat's **Substack Posts Scraper**. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

#### Choose your assistant

- **Claude and Claude Code (Anthropic):** [Apify setup guide](https://docs.apify.com/integrations/claude).
- **ChatGPT and Codex (OpenAI):** [Apify setup guides](https://docs.apify.com/integrations/openai).
- **Grok Bot (xAI):** [Apify plugin guide](https://docs.apify.com/integrations/grok-bot).
- **Gemini CLI (Google):** [MCP server setup guide](https://geminicli.com/docs/tools/mcp-server/). Use the server URL below with the HTTP configuration supported by Gemini CLI.

#### Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/substack-posts-scraper"
    }
  }
}
```

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

#### Use Apify's general server

Already connected to the general Apify MCP server at `https://mcp.apify.com`? You can use it to find this scraper without adding another server. Ask your assistant to use **Substack Posts Scraper** by Fetch Cat, or give it the identifier `fetch_cat/substack-posts-scraper`.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

#### Example requests

- “Collect five recent public posts from Lenny’s Newsletter and give me a table of titles, dates and source links.”
- “Get ten posts each from Import AI and One Useful Thing, then summarize the available public previews by topic.”
- “Collect posts from my publication list published during September 2026 and prepare a spreadsheet of the available authors and engagement counts.”

Normal Apify charges apply. Check the [live Pricing tab](https://apify.com/fetch_cat/substack-posts-scraper/pricing) before asking your assistant to run a larger batch.

### Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through [x402](https://docs.apify.com/integrations/x402). This experimental option lets an agent use Apify without a conventional account.

[Apify AGI](https://agi.apify.com/) provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

#### Other ways for agents to pay

Apify also supports **MPP** for prepaid access through AGI and **Skyfire** for eligible Actors. If your agent already uses one of these, see the [agentic payment guide](https://docs.apify.com/agents/payments) to choose the right setup.

### FAQ

#### Can I export Substack posts without logging in?

Yes, this Actor uses public publication data. Availability varies and it cannot retrieve content hidden behind a subscriber login.

#### Can I download complete articles?

Only when public responses expose the content. `bodyText` may contain a preview, and enabling `includeBodyHtml` cannot bypass a paywall.

#### Is this a Substack API alternative?

It provides structured public post data through Apify’s API. It is not Substack’s official API and does not support account management, subscriber data or global publication discovery.

#### Can I search all Substack publications?

No. Supply publication URLs first. The optional search term applies within each publication’s archive, and feed fallback does not preserve that search filter.

#### Why did my run return no posts?

Check publication URLs, date filters and the run summary. A feed can contain fewer posts than an archive. No saved posts currently produces a failed run, not a guarantee that the target has no content.

#### Can I automate CSV or Excel exports?

Yes. Use the dataset export options or retrieve records through the API and add file creation to your workflow.

### Related Actors

- [Substack Notes Scraper](https://apify.com/fetch_cat/substack-notes-scraper) — Public short-form Notes and engagement.
- [Hacker News Search Scraper](https://apify.com/fetch_cat/hacker-news-search-scraper) — Search technology stories and comments.
- [Telegram Channel Posts Scraper](https://apify.com/fetch_cat/telegram-channel-posts-scraper) — Public channel content.
- [Product Hunt Scraper](https://apify.com/fetch_cat/product-hunt-scraper) — Product launches and discussions.
- [Google News Scraper](https://apify.com/fetch_cat/google-news-scraper) — News headlines and publishers.

### Support and feedback

For custom additions or integrations, email <hello@fetchcat.dev>.

Found an error or have a feature request? [Open an issue](https://apify.com/fetch_cat/substack-posts-scraper/issues) with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See [Apify’s guide to sharing a run when reporting an issue](https://docs.apify.com/account/collaboration/general-resource-access#automatically-sharing-runs-via-actor-issues). Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. [Leave a review from the Actor page](https://apify.com/fetch_cat/substack-posts-scraper) when that option is available to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥

# Changelog

This Actor's version history is a separate document: https://apify.com/fetch_cat/substack-posts-scraper/changelog.md

# Actor input Schema

## `publicationUrls` (type: `array`):

Substack publication homepages, custom domains, or bare domains to scrape (for example, https://www.lennysnewsletter.com or https://example.substack.com).

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to save from each publication.

## `searchTerm` (type: `string`):

Optional search within each supplied publication archive, not global Substack search. Leave empty for newest archive posts. Feed fallback does not apply this search term, so fallback records may not match it.

## `dateFrom` (type: `string`):

Optional inclusive UTC start date (YYYY-MM-DD). Posts with a usable date before it are skipped. Posts with missing or unparseable dates are retained.

## `dateTo` (type: `string`):

Optional inclusive UTC end date (YYYY-MM-DD). Posts with a usable date after it are skipped. Must not be earlier than Published after. Posts with missing or unparseable dates are retained.

## `includeBodyHtml` (type: `boolean`):

Include public body HTML when present. Default disabled for smaller datasets. Public body text or preview text can still be saved when this is disabled. Enabling it cannot bypass a paywall.

## `resumePublicationUrls` (type: `array`):

Optional checkpoint URLs copied from RUN_SUMMARY after a run stopped before its deadline. These are added before publicationUrls without changing existing inputs.

## `includeRssFallback` (type: `boolean`):

Default enabled. If an archive fails or returns no matching posts, try the public feed. Date filters still apply to usable dates, but the search term does not apply to feed results. Feeds may contain only recent posts and fewer metadata fields. This does not unlock paid content.

## `concurrency` (type: `integer`):

Number of publications processed in parallel. Keep low for reliable public archive access.

## Actor input object example

```json
{
  "publicationUrls": [
    {
      "url": "https://www.lennysnewsletter.com"
    }
  ],
  "maxPostsPerPublication": 20,
  "searchTerm": "",
  "includeBodyHtml": false,
  "includeRssFallback": true,
  "concurrency": 3
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        {
            "url": "https://www.lennysnewsletter.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fetch_cat/substack-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publicationUrls": [{ "url": "https://www.lennysnewsletter.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("fetch_cat/substack-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    {
      "url": "https://www.lennysnewsletter.com"
    }
  ]
}' |
apify call fetch_cat/substack-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fetch_cat/substack-posts-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6x7OQTaAafm6aRfaZ/builds/zYpb6TiG0LwdUhhDD/openapi.json