Hacker News Scraper — Stories, Comments, Jobs & Users

Hacker News Scraper 🚀 — Stories, Comments, Jobs & User Profiles Extractor

Extract stories, Show HN posts, Ask HN discussions, full comment threads, monthly "Who's Hiring" tech job listings, and user karma profiles from Hacker News at scale. This Hacker News scraper gives you access to the pulse of the tech ecosystem with zero API key constraints, no rate-limit hurdles, and complete historical depth.

Search by keywords or tags, apply strict score thresholds (minPoints, minComments), unpack comment trees from specific story IDs, or fetch developer profiles. With deep automated pagination unlocked, you can harvest 1,000+ stories or comments per query directly into JSON, CSV, or Excel.

---

⚡ At a glance

Feature	Detail
Target Website	Hacker News (Y Combinator Tech Community)
Data Scope	Stories, Show HN, Ask HN, Front Page, Comments, Who's Hiring Jobs, User Profiles
Output Formats	JSON, CSV, Excel (XLSX), XML, HTML Table, RSS
Key Data Fields	Title, Story URL, Author, Points/Score, Comments Count, Story/Comment Text, Created Date, HN URL
Search Capabilities	Keyword search, tag filtering, point & comment thresholds, sorting (popularity vs. newest)
Special Modes	Unpack full comment threads (e.g. Who is Hiring? job posts) & User profile enrichment
Pagination & Depth	Unlocked 1,000+ multi-page traversal (100 hits per page)
Authentication	None required (No cookies, no API key, no HN account needed)
Average Speed	~300–500 items in under 10 seconds
Integrations	Webhooks, Zapier, Make, Google Sheets, Airbyte, Python, Node.js, cURL

---

🔎 Why scrape Hacker News?

Hacker News is the premier watercooler for the global engineering, open-source, and startup ecosystem. Because submissions are curated and upvoted by founders, principal engineers, researchers, and tech leaders, HN serves as an early indicator for industry-defining trends:

• Developer & Tech Trend Detection: Monitor which programming languages (Rust, Zig, Mojo), AI frameworks, or cloud databases are capturing mindshare.
• Developer Recruitment & Sourcing: Every month, the viral Ask HN: Who is hiring? thread generates hundreds of direct job openings posted by hiring managers.
• Product Launch Intelligence: Track Show HN submissions to monitor emerging competitor products and evaluate authentic community reception.
• Sentiment & Quantitative NLP: Mine unfiltered developer feedback, technical criticism, and brand perceptions on major corporate announcements.

Because the native website is formatted strictly for reading with no bulk export or granular filters, this Hacker News data extractor provides a standardized, fast pipeline for automated analysis.

---

💼 Core Use Cases & Applications

1. "Who is Hiring?" Job Mining & Tech Recruiting

Extract every comment from the monthly Ask HN: Who is hiring? thread. Automatically convert free-form comments into structured recruiting leads containing company names, remote policies, tech stacks, and direct email contacts.

2. DevRel & Brand Sentiment Monitoring

Track mentions of your developer tools, open-source repositories, or SaaS products across all stories and comment threads. Benchmark engagement using upvote scores (points) and discussion velocity (numComments).

3. Startup & VC Market Intelligence

Monitor Show HN launches and high-point submissions (points >= 200) to identify fast-growing startups, novel architecture paradigms, and seed-stage investments before they hit mainstream tech news.

4. Automated Tech Newsletters & Slack Bots

Build automated alerts that ping your engineering or executive teams whenever a topic relevant to your business reaches the Hacker News front page.

5. Academic & LLM Training Datasets

Compile high-quality developer discussions, technical debates, and programming tutorials for fine-tuning NLP models, sentiment classifiers, and domain-specific language models.

---

🏷️ Supported Tags & Content Types

Filter your searches using native Hacker News taxonomy:

• 📰 story — General external articles, essays, and technical writeups
• 🚀 show_hn — Community-submitted projects, open-source tools, and startup launches
• ❓ ask_hn — Questions posed to the tech community (including hiring and freelancing threads)
• 🏆 front_page — Stories currently or recently featured on the Hacker News homepage
• 🗳️ poll — Community polls and surveys
• 💬 comment — Individual comments across all discussions

---

📊 Extracted Data Fields

The scraper outputs normalized, typed JSON records. Depending on whether you are querying stories, comments, or users, the appropriate fields are populated:

📰 Story Fields (type: "story")

Field Name	Type	Coverage	Description
id	String	🟢 High (100%)	Unique Hacker News item identifier (e.g. 41982734)
type	String	🟢 High (100%)	Record type: story
title	String	🟢 High (100%)	Story headline
url	String	🟢 High (88%)	External article URL (empty for self-posts)
author	String	🟢 High (100%)	Username of the submitter
points	Number	🟢 High (100%)	Total upvotes / community score
numComments	Number	🟢 High (100%)	Total number of discussion replies
storyText	String	🟡 Conditional (40%)	Cleaned text body for Ask HN and text submissions
createdAt	String	🟢 High (100%)	ISO-8601 submission timestamp
tags	Array of Strings	🟢 High (100%)	Internal tags attached to the submission
hnUrl	String	🟢 High (100%)	Direct URL to discussion thread on Hacker News
scrapedAt	String	🟢 High (100%)	ISO-8601 extraction timestamp

---

💬 Comment Fields (type: "comment")

Field Name	Type	Coverage	Description
id	String	🟢 High (100%)	Unique comment item identifier
type	String	🟢 High (100%)	Record type: comment
author	String	🟢 High (100%)	Comment author username
commentText	String	🟢 High (100%)	Full text with HTML tags stripped and entities decoded
points	Number	🟡 Conditional (30%)	Upvote score (if available)
storyId	Number	🟢 High (100%)	Parent story ID
storyTitle	String	🟢 High (95%)	Title of the parent story discussion
parentId	Number	🟢 High (100%)	Direct parent comment or story ID
createdAt	String	🟢 High (100%)	ISO-8601 comment timestamp
hnUrl	String	🟢 High (100%)	Direct link to the individual comment permalink

---

👤 User Profile Fields (type: "user")

Field Name	Type	Coverage	Description
id	String	🟢 High (100%)	Hacker News username
type	String	🟢 High (100%)	Record type: user
karma	Number	🟢 High (100%)	Total cumulative user karma
about	String	🟡 Conditional (65%)	User bio / about description
submittedCount	Number	🟢 High (100%)	Total number of lifetime submissions
createdAt	String	🟢 High (100%)	User account creation timestamp
hnUrl	String	🟢 High (100%)	Direct URL to user profile page

---

⚙️ Input Parameters

Configure the actor easily using the Apify Console or API:

Parameter	Type	Default	Description
searchTerms	Array of Strings	["rust"]	List of keywords to search (e.g. ["rust", "ai", "kubernetes"]). Leave empty to scrape by tag only.
tags	String (Select)	"story"	Filter by tag: story, show_hn, ask_hn, front_page, poll, or comment.
sortBy	String (Select)	"popularity"	Sort order: popularity (most points & relevance) or date (newest first).
minPoints	Integer	0	Minimum score threshold (e.g. set to 100 for viral stories).
minComments	Integer	0	Minimum comment count threshold (e.g. set to 50 for heavily discussed posts).
storyIds	Array of Strings	[]	Specific story IDs to extract comments from (e.g. monthly Who is hiring? threads).
users	Array of Strings	[]	List of usernames to fetch profile dossiers and karma for.
maxItems	Integer	0	Total maximum items to retrieve across all queries (0 = unlimited). Prefilled at 100.
maxPagesPerQuery	Integer	0	Maximum pages per query (0 = unlimited, governed by maxItems). Each page yields 100 hits.
proxyConfiguration	Object	Datacenter	Apify proxy configuration. Datacenter proxies deliver optimal speed and cost efficiency out of the box.

---

📦 Example Output

Here is a representative JSON record from a story query:

{
  "id": "41982341",
  "type": "story",
  "title": "Show HN: Fast SQLite replication engine written in Rust",
  "url": "https://github.com/example/fast-sqlite-repl",
  "author": "techfounder",
  "points": 342,
  "numComments": 87,
  "storyText": "Hi HN! We built a lightweight consensus-free replication protocol for SQLite...",
  "createdAt": "2026-11-08T18:42:10.000Z",
  "tags": ["story", "show_hn", "author_techfounder"],
  "hnUrl": "https://news.ycombinator.com/item?id=41982341",
  "scrapedAt": "2026-11-10T13:35:00.000Z"
}

---

🔄 Comparison: Hacker News Scraper vs. Alternatives

Feature / Capability	Hacker News Scraper	Manual Scraping / Beautiful Soup	Generic Web Scraper
Speed & Throughput	~500 items / 10 seconds	Slow and rate-limited	Heavy browser overhead
Comment Thread Extraction	Automated recursive comment flattening	Fragile DOM traversal	Breaks on nested trees
"Who is Hiring" Support	Direct story comment extraction	Tedious copy-pasting	Incomplete pagination
1,000+ Deep Pagination	Unlocked deep Algolia API pagination	Blocked after ~100 items	Timeout errors
De-duplication	In-memory ID tracking prevents duplicates	High duplicate rate	High duplicate rate
HTML Sanitization	Decodes entities & strips markup	Manual regex cleanup	Leaves messy HTML

---

💻 How to Use via API

You can trigger runs programmatically using the Apify API and official SDKs:

Python Example

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Search top Rust & AI stories with min 100 points
run_input = {
    "searchTerms": ["rust", "deepseek"],
    "tags": "story",
    "sortBy": "popularity",
    "minPoints": 100,
    "maxItems": 200,
    "maxPagesPerQuery": 0
}

run = client.actor("haketa/hackernews-scraper").call(run_input=run_input)

# Fetch dataset items
items = client.dataset(run["defaultDatasetId"]).list_items().items
for story in items:
    print(f"[{story['points']} pts] {story['title']} - {story['hnUrl']}")

Node.js Example (Extract Who's Hiring Jobs)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

// Extract all job postings from a specific Ask HN: Who is hiring? thread ID
const input = {
    storyIds: ['41950000'], // Replace with active thread ID
    maxItems: 500,
    maxPagesPerQuery: 0
};

const run = await client.actor('haketa/hackernews-scraper').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

console.log(`Successfully scraped ${items.length} job postings!`);

cURL Example

curl "https://api.apify.com/v2/acts/haketa~hackernews-scraper/runs?token=YOUR_APIFY_TOKEN" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "searchTerms": ["show hn"],
    "tags": "show_hn",
    "sortBy": "date",
    "maxItems": 100
  }'

---

📈 Analyzing Hacker News Trends with Pandas

Load the extracted stories into Python Pandas to uncover top authors, domain distributions, and discussion trends:

import pandas as pd
from urllib.parse import urlparse

# Load dataset CSV exported from Apify
df = pd.read_csv("hackernews_stories.csv")

# Extract domain from external URLs
def extract_domain(url):
    if pd.isna(url) or not str(url).startswith("http"):
        return "self-post"
    return urlparse(str(url)).netloc.replace("www.", "")

df["domain"] = df["url"].apply(extract_domain)

print("Top 10 Most Shared Domains:")
print(df["domain"].value_counts().head(10))

# Average points by submission tag
print("\nScore Statistics:")
print(df[["points", "numComments"]].describe())

# Top viral stories
top_stories = df.sort_values(by="points", ascending=False).head(5)
print("\nTop 5 Stories by Upvotes:")
print(top_stories[["title", "points", "numComments", "author"]])

---

🗄️ Database Storage: SQL / PostgreSQL Schema

For tech intelligence teams archiving Hacker News items in a relational database:

CREATE TABLE hn_items (
    id VARCHAR(32) PRIMARY KEY,
    item_type VARCHAR(16) NOT NULL, -- story | comment | user
    title TEXT,
    url TEXT,
    author VARCHAR(64),
    points INTEGER,
    num_comments INTEGER,
    story_text TEXT,
    comment_text TEXT,
    story_id VARCHAR(32),
    parent_id VARCHAR(32),
    created_at TIMESTAMPTZ,
    hn_url TEXT NOT NULL,
    scraped_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_hn_type_points ON hn_items (item_type, points DESC);
CREATE INDEX idx_hn_author ON hn_items (author);
CREATE INDEX idx_hn_created ON hn_items (created_at DESC);

---

💡 Best Practices & Pro Tips

1. Filtering for Viral Hits: When monitoring competitor launches or high-impact essays, set minPoints: 100 and minComments: 30. This filters out low-signal submissions automatically.
2. Finding the Latest "Who is Hiring?" Thread: Search Ask HN: Who is hiring? with sortBy: "date" and tags: "ask_hn" to find the most recent monthly thread ID, then pass that ID into storyIds.
3. Empty Run Exploration: Running with default {} automatically crawls current top stories on Hacker News with zero configuration.
4. Deep Historical Crawling: Setting maxPagesPerQuery: 0 removes all per-query pagination limits. The actor will paginate through hundreds of pages (100 items per page) until reaching your maxItems target.
5. Deduplication: In-memory {type}:{id} tracking ensures that even if an item appears in multiple search queries, it is only pushed once.

---

❓ Frequently Asked Questions (FAQ)

Do I need a Hacker News account or API key?

No. The scraper accesses public Hacker News endpoints without requiring any account login, cookies, or developer keys.

How many comments can I extract from a single story?

You can extract all comments on any thread, including massive monthly hiring posts with 800+ replies. Set storyIds to the target thread ID and configure maxItems: 0.

Can I fetch user karma and profiles?

Yes. Pass usernames into the users parameter (e.g. ["dang", "sama", "pg"]) to retrieve current karma, account registration dates, and bio text.

Does the scraper run on Datacenter proxies?

Yes. Hacker News public data endpoints are fully accessible via Apify's standard Datacenter proxies, keeping execution costs minimal.

Can I export to Google Sheets or Excel?

Yes. From the Apify Console or API, you can export your dataset directly as an .xlsx file, .csv, or automatically synchronize with Google Sheets using Apify integrations.

---

💼 Parsing "Who is Hiring" Job Comments

When mining monthly hiring threads, comment text usually begins with a structured company header. Here is a Python helper to parse role details and remote availability:

import re

def parse_job_comment(text):
    first_line = text.split("\n")[0]
    is_remote = bool(re.search(r"\b(remote|anywhere|wfh)\b", text, re.I))
    has_visa = bool(re.search(r"\b(visa|sponsor)\b", text, re.I))
    
    # Extract email addresses for direct candidate application
    emails = re.findall(r"[\w\.-]+@[\w\.-]+\.\w+", text)
    
    return {
        "header": first_line[:120],
        "remote_friendly": is_remote,
        "sponsors_visa": has_visa,
        "contact_emails": emails
    }

---

⚖️ Disclaimer

This actor is an independent open-source tool developed for research, analytics, and intelligence purposes. It is not affiliated with, endorsed by, or sponsored by Hacker News, Y Combinator, or Algolia.

---

📋 Changelog

Version 1.2 (2026-11)

• Deep 1,000+ Pagination: Set maxPagesPerQuery default to 0 (minimum: 0) and enabled deep multi-page traversal (100 items/page) governed by maxItems.
• Enhanced Entity Decoding: Improved HTML entity cleanup (/, ', ", &) and sanitized comment formatting.
• Optimized Prefill Execution: Fast prefill configuration completing in under 10 seconds.
• Expanded Documentation: Complete field coverage indicators, recruiting workflow tutorial, SQL schema, and pandas analytics recipes.

Version 1.1

• Added comment extraction for specific story IDs (Who's Hiring support).
• Added user profile and karma extraction via Firebase.
• Mirrored output to Key-Value Store OUTPUT record for synchronous calls.

Version 1.0

• Initial public release supporting keyword search, tag filtering, points thresholding, and multi-format exports.