Substack Scraper – Newsletters, Creators, Posts & Leads

Substack Scraper — Newsletters, Creators, Posts & Leads

The ultimate Substack intelligence and creator lead extraction tool. This Substack scraper extracts data from Substack without login or API keys. Discover top newsletters across any category — complete with verified author identities, subscriber counts, monetization status, and contact details — and retrieve complete publication post archives including full plain-text articles, engagement metrics (reactions, comments, restacks), and podcast audio feeds. Export seamlessly to JSON, CSV, Excel, or API.

Engineered for growth marketers, sponsorship sales reps, PR and influencer agencies, media analysts, competitive intelligence researchers, and AI/ML engineers feeding long-form text into LLMs and RAG pipelines.

---

⚡ At a glance

Feature	Specification
Data source	Substack global platform directory and public publication archives
Output formats	JSON, CSV, Excel (XLSX), XML, RSS, HTML table
Pagination model	Continuous pagination (unlocks 1,000+ publications and historical posts)
Throughput speed	~20–50 posts or newsletter profiles per second
Dual record types	Publication lead records (recordType: "publication") & Post records (recordType: "post")
Subscriber coverage	~96%+ clean numerical subscriber counts extracted from leaderboard metadata
Content extraction	Full cleaned article plain-text, markdown structure, optional raw HTML
Podcast audio support	Extracts direct audio stream links and episode duration
Proxy requirements	Datacenter proxies supported (lightweight HTTP requests, no browser required)

---

🔎 Why scrape Substack?

Substack hosts hundreds of thousands of independent journalists, industry analysts, software engineers, financial researchers, and thought leaders. Its category leaderboards represent the most authoritative, ranked directory of high-influence creators available on the modern web.

However, Substack lacks any official public API or bulk export tooling. Extracting creator contact information or compiling post archives manually requires endless clicking, inspecting individual author bios, and copying engagement numbers into spreadsheets.

This Substack scraper bridges this gap. By querying Substack's native endpoints, the actor captures ranked creator rosters across 20+ verticals, resolves subscriber magnitude counts into clean sortable numbers, and downloads full publication archives with complete engagement statistics.

High-Value Strategic Applications

• 🧲 Creator & Influencer Lead Generation: Identify top newsletters in finance, technology, AI, healthcare, and crypto for sponsored placement, newsletter takeovers, and podcast ad sponsorships.
• 📊 Competitive Benchmarking: Analyze competing industry newsletters to track post frequency, audience engagement ratios (reactions per subscriber), and paid tier monetization adoption.
• 🎙️ PR & Media Pitching: Build highly targeted media lists of active journalists and subject matter experts covering specific macroeconomic, venture, or cultural themes.
• 🤖 LLM Training & Knowledge Retrieval (RAG): Ingest clean, long-form expert essays and domain analyses into vector databases, semantic search engines, and fine-tuning pipelines.
• 📰 Executive Content Monitoring: Monitor key thought leaders or competitor publications on a recurring schedule to surface breaking industry opinions and emerging commentary.
• 💵 Creator Economy Market Research: Quantify the growth trajectory of paid publishing models, bestseller badge distributions, and independent publishing economics.

---

✨ Features & Capabilities

• ✅ Category Leaderboard Discovery: Browse 20+ categories (Technology, Business, Finance, Crypto, Science, News, Culture, Politics, Health, etc.) and extract ranked creator directories.
• ✅ Clean Numerical Subscriber Counts: Normalizes vague phrases ("Over 1,100,000 subscribers") into clean numerical values (1100000) for instant spreadsheet sorting and threshold filtering.
• ✅ Full Article Text Extraction: Captures entire post bodies stripped of boilerplate HTML, formatting code, and navigation elements.
• ✅ Granular Engagement Metrics: Retrieves reaction (like) counts, public comment counts, and Substack platform restack (re-share) counts.
• ✅ Monetization & Badge Tracking: Identifies paid subscription paywalls, Founding Member options, and Substack Bestseller badges.
• ✅ Podcast Media Metadata: Detects audio episodes, extracting episode enclosure URLs and duration timings.
• ✅ Custom Domain Support: Seamlessly follows redirects and resolves custom domains (e.g. astralcodexten.substack.com → astralcodexten.com).
• ✅ Zero-Config Instant Demo: Runs out of the box with default category sampling without requiring manual URL inputs.

---

📥 Input Parameters

Parameter	Type	Default	Description
discoverCategories	Array	[]	Category IDs to discover top newsletters (e.g., "4" for Tech, "62" for Business, "153" for Finance).
publications	Array	[]	List of publication subdomains (e.g., "astralcodexten") or URLs.
postUrls	Array	[]	Direct Substack post URLs (.../p/...) to scrape individually.
scrapePosts	Boolean	true	When true, retrieves posts from specified publications. When false, only outputs publication profile leads.
includePostContent	Boolean	true	Extract full article text for each post. Turn off for metadata-only scraping.
includeHtml	Boolean	false	Include raw HTML content in the dataset.
audience	String	"all"	Filter posts by access level ("all", "free", "paid").
maxPostsPerPublication	Integer	0	Max posts per publication. Set to 0 for unlimited historical crawl. Prefill 20.
maxPublications	Integer	0	Max publications to collect during category discovery. Set to 0 for unlimited. Prefill 50.
sinceDate	String	""	ISO date string (e.g. 2026-01-01). Only scrapes posts published on or after this date.
proxyConfiguration	Object	{"useApifyProxy": true}	Apify Proxy settings (datacenter proxies provide top performance).

---

📤 Output Data Fields

The scraper outputs two distinct record types depending on your run configuration: Publication Leads (recordType: "publication") and Post Records (recordType: "post").

📰 Publication Lead Record Fields

Field	Type	Coverage	Description
recordType	String	🟢 High	Record identifier: "publication".
name	String	🟢 High	Title of the publication (e.g., The Pragmatic Engineer).
author	String	🟢 High	Primary creator or publisher display name.
subscribers	String	🟢 High	Clean numerical subscriber count (e.g., 773,000).
subscribersLabel	String	🟢 High	Raw descriptive subscriber badge text.
contactEmail	String	🟡 Conditional	Public contact email address when provided by the author.
category	String	🟢 High	Category vertical name (e.g., Technology, Finance).
tagline	String	🟢 High	Publication description and mission tagline.
isMonetizing	Boolean	🟢 High	Boolean flag indicating active paid subscription tiers.
hasPodcast	Boolean	🟢 High	Boolean flag indicating whether the publication hosts podcast audio.
authorBestsellerTier	String	🟡 Conditional	Substack badge status (e.g., Hundreds of thousands of paid subscribers).
url	String	🟢 High	Canonical web URL of the publication.
scrapedAt	String	🟢 High	UTC timestamp of record creation.

📝 Post Record Fields

Field	Type	Coverage	Description
recordType	String	🟢 High	Record identifier: "post".
title	String	🟢 High	Headline title of the newsletter post.
subtitle	String	🟡 Conditional	Post deck or subtitle summary.
publication	String	🟢 High	Name of the hosting newsletter.
author	String	🟢 High	Byline author of the post.
postDate	String	🟢 High	ISO 8601 publication timestamp.
audience	String	🟢 High	Access tier: "free", "paid", or "founding".
reactionCount	Number	🟢 High	Total likes/reactions registered on the post.
commentCount	Number	🟢 High	Count of public comments submitted on the post.
restackCount	Number	🟢 High	Number of Substack restacks (shares across Notes).
wordcount	Number	🟢 High	Calculated word count of the article text.
content	String	🟢 High	Full plain-text post content (markdown formatted).
audioUrl	String	🟡 Conditional	Direct MP3 streaming URL for podcast episodes.
duration	String	🟡 Conditional	Duration of the attached audio episode.
url	String	🟢 High	Direct permalink to the post.
scrapedAt	String	🟢 High	UTC timestamp of record extraction.

---

💻 Sample Output Records

Publication Lead Example:

{
  "recordType": "publication",
  "name": "Pragmatic Engineer",
  "subdomain": "pragmaticengineer",
  "url": "https://newsletter.pragmaticengineer.com",
  "author": "Gergely Orosz",
  "tagline": "The #1 technology newsletter on Substack. Big tech and high-growth startups, from the inside.",
  "subscribers": "850,000",
  "subscribersLabel": "Over 850,000 subscribers",
  "contactEmail": "gergely@pragmaticengineer.com",
  "category": "Technology",
  "isMonetizing": true,
  "hasPodcast": true,
  "authorBestsellerTier": "Tens of thousands of paid subscribers",
  "scrapedAt": "2026-11-10T14:40:00.000Z"
}

Post Record Example:

{
  "recordType": "post",
  "title": "State of Software Engineering in 2026",
  "subtitle": "Hiring trends, AI code generation in production, and compensation benchmarks",
  "publication": "Pragmatic Engineer",
  "author": "Gergely Orosz",
  "postDate": "2026-10-24T12:00:00.000Z",
  "audience": "free",
  "reactionCount": 1420,
  "commentCount": 215,
  "restackCount": 89,
  "wordcount": 3480,
  "content": "Over the past six months, engineering organizations have reached a pivotal inflection point...",
  "audioUrl": "https://api.substack.com/media/audio/1849201.mp3",
  "duration": "00:42:15",
  "url": "https://newsletter.pragmaticengineer.com/p/state-of-software-engineering-2026",
  "scrapedAt": "2026-11-10T14:40:05.000Z"
}

---

🛠️ Step-by-Step API Integration

Integrate Substack intelligence into your automated workflows via Python, Node.js, or cURL.

Python Integration

from apify_client import ApifyClient

# Initialize Apify client
client = ApifyClient("YOUR_APIFY_TOKEN")

# Discover top Finance newsletters for sponsorship outreach
run_input = {
    "discoverCategories": ["153"],  # 153 = Finance
    "scrapePosts": False,            # Only return publication lead profiles
    "maxPublications": 100
}

# Run the actor
run = client.actor("haketa/substack-scraper").call(run_input=run_input)

# Print discovered creators
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"[{item.get('subscribers')} subs] {item.get('name')} by {item.get('author')} ({item.get('contactEmail')})")

Node.js Integration

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: 'YOUR_APIFY_TOKEN',
});

const runInput = {
    publications: ['astralcodexten'],
    maxPostsPerPublication: 25,
    includePostContent: true,
};

const run = await client.actor('haketa/substack-scraper').call(runInput);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

console.log(`Fetched ${items.length} posts.`);
items.forEach(post => {
    console.log(`${post.title} (${post.postDate}) - ${post.reactionCount} likes`);
});

cURL Execution

curl "https://api.apify.com/v2/acts/haketa~substack-scraper/runs?token=YOUR_APIFY_TOKEN" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "discoverCategories": ["4"],
    "maxPublications": 50,
    "scrapePosts": false
  }'

---

📚 Complete Category Directory Reference

Select any of the following official Substack category identifiers when setting discoverCategories to target specific editorial niches:

Category ID	Category Name	Typical Content Focus & Lead Demographics
4	Technology	AI developments, engineering leadership, software architecture, developer tools
62	Business	Venture capital, startup operating playbooks, SaaS metrics, management
153	Finance	Stock market analysis, macroeconomic forecasting, value investing, commodities
118	Crypto	Blockchain infrastructure, DeFi protocols, Web3 ecosystems, digital assets
134	Science	Biotech, quantum physics, space exploration, longevity and climate tech
103	News	Independent investigative reporting, foreign policy, geopolitical commentary
96	Culture	Digital culture, sociology, trend forecasting, modern internet aesthetics
76739	U.S. Politics	Electoral analysis, legislative reporting, political commentary, policy debate
355	Health & Wellness	Evidence-based nutrition, fitness science, mental health, biohacking
34	Education	Higher ed insights, pedagogical methods, homeschool resources, lifelong learning
11	Music	Album reviews, music industry economics, artist essays, independent releases
109	Travel	Destination guides, remote work dispatch, nomadic lifestyle, regional travel
18	History	Historical deep dives, military strategy, archival retrospectives, biography
114	Philosophy	Ethics, epistemology, classical philosophy, contemporary rationalist thought
223	Faith & Spirituality	Theological discourse, contemplative traditions, spiritual literature
339	Literature	Fiction essays, literary criticism, author interviews, serialized novels
94	Sports	Sports analytics, team commentary, coaching breakdowns, endurance sports
13645	Food & Drink	Culinary techniques, restaurant reviews, cocktail recipes, food industry trends
15414	Climate & Environment	Clean energy transition, environmental policy, ESG analysis, biodiversity
49692	Humor	Satirical commentary, humor essays, comedic serialized writing, cartoons

---

🔄 Automated Workflows & CRM Integrations

Automate your creator research and content pipelines by integrating the Substack Scraper into your modern data stack:

flowchart LR
    A[Substack Scraper] -->|Lead Records| B[Apify Dataset Webhook]
    B --> C{Automation Engine}
    C -->|Creator Leads| D[Airtable / HubSpot CRM]
    C -->|New Article Text| E[OpenAI / Anthropic LLM]
    E --> F[Vector DB / Pinecone / Qdrant]

Popular Integrations:

1. Sponsorship Pipeline into Airtable & Notion: Automatically push newly discovered publications into an Airtable outreach base. Filter by subscribers > 50,000 to alert partnership managers when a fast-growing publication emerges.

2. RAG & Knowledge Base Synchronization: Trigger an automated embedding workflow whenever an author publishes a new post. The clean plain-text content field is ready for chunking without HTML stripping or regex sanitization.

3. Slack Real-Time High-Engagement Post Alerts: Configure a webhook to notify editorial teams on Slack whenever monitored publications release articles that surpass 1,000 reactions or 200 comments within 24 hours.

---

💡 Lead Generation & Scraping Tips

• Fast Creator Lead Generation: To build sponsorship or PR lists quickly, select multiple target categories in discoverCategories and toggle scrapePosts: false. This pulls hundreds of high-profile publication profiles with subscriber counts and emails in seconds without downloading historical post archives.
• RAG & Vector Database Preparation: When preparing documents for LLM ingestion, leave includePostContent: true and includeHtml: false. The actor's built-in text sanitizer eliminates header/footer noise, providing clean paragraph blocks with word count metrics.
• Incremental Crawling: Use sinceDate (e.g., "2026-11-01") on scheduled weekly runs to only capture posts published after your last update, saving storage and execution time.

---

❓ Frequently Asked Questions (FAQ)

Does this scraper require a Substack login or account? No login, credentials, or cookies are required. The actor gathers data strictly from public leaderboards, public author profiles, and freely available posts.

Can I scrape paywalled / paid subscriber posts? For posts restricted to paid subscribers, Substack serves only a public preview. The actor extracts all publicly readable text, post titles, dates, author bylines, and engagement counts. Full private paywalled content is protected behind authentication.

How accurate are the subscriber counts? Substack displays subscriber counts on category leaderboards and profile banners as tier magnitudes (e.g., "Over 500,000 subscribers" or "500K+"). The actor extracts and normalizes this figure into a clean numeric string (500,000), allowing direct programmatic sorting.

What categories are available for discovery? Over 20 distinct categories are supported, including Technology (4), Business (62), Finance (153), Crypto (118), Science (134), News (103), Politics (76739), Health (355), Education (34), Travel (109), History (18), Food & Drink (13645), and Climate (15414).

How does the scraper handle custom domains? Many prominent Substack publications operate on independent domain names (e.g. pragmaticengineer.com rather than pragmaticengineer.substack.com). The scraper automatically resolves domain redirects and fetches publication metadata seamlessly.

Can I scrape individual post URLs directly? Yes. If you have specific articles you wish to analyze (e.g. https://newsletter.pragmaticengineer.com/p/state-of-software-engineering-2026), supply them directly in postUrls. The scraper extracts the full article content, metrics, and author details immediately without crawling the rest of the publication.

How does the scraper handle audio podcasts on Substack? For podcast publications, Substack attaches MP3 enclosures and episode runtimes. The actor captures the direct audioUrl and formatted duration timestamp alongside the show notes text.

What is the recommended concurrency setting? A maxConcurrency setting between 4 and 8 provides the optimal balance of high-throughput extraction and network stability when running on Apify Proxy.

How frequently should I schedule publication monitoring? Most active newsletters publish 1–3 times per week. Scheduling a run to trigger twice weekly with sinceDate set to 3 days prior will capture all new editorial releases and updated comment counts with minimal compute consumption.

Can I export data directly to cloud storage (Amazon S3, Google Cloud Storage)? Yes. You can use Apify's native cloud storage integrations or trigger an AWS Lambda function via dataset webhooks upon run completion.

---

🔗 Related Creator & Content Scrapers

• Google Maps Scraper — Extract business leads, phones, and websites across any geographic radius.
• Trip.com Reviews Scraper — Hotel reviews, AI translations, and customer ratings.
• Trustpilot Reviews Scraper — Trust ratings and verified customer feedback.
• Turo Scraper — Car sharing listings, host leads, and vehicle availability.

---

📋 Changelog

• v1.2 (2026-11)
  • Unlocked unlimited 1,000+ deep pagination across category leaderboards and publication archives.
  • Standardized zero-config demo execution with fast Technology leaderboard sampling.
  • Enhanced subscriber count normalization and custom domain redirect resolution.
  • Optimized pure HTTP concurrency for sub-30s execution speeds.
• v1.1 (2026-08)
  • Added podcast enclosure audio URL and episode duration extraction.
  • Implemented Substack platform restack and comment counter parsing.
• v1.0 (2026-07)
  • Initial public launch of Substack newsletter, creator, and post content scraper.