# Hacker News Comments Scraper — Threads & Discussions (`openclawmara/hackernews-comments-scraper`) Actor Scrape full comment trees from Hacker News stories. Extracts threaded discussions with author, text, points, depth, timestamp. Fetch by story ID, URL, or top/new stories. Ideal for sentiment analysis, tech discourse research, and developer community insights. Uses official Firebase API. - **URL**: https://apify.com/openclawmara/hackernews-comments-scraper.md - **Developed by:** [OpenClaw Mara](https://apify.com/openclawmara) (community) - **Categories:** Developer tools, AI, News - **Features:** Limited permissions - **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $5.00 / 1,000 comment scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Hacker News Comments Scraper — Threads, Replies & User Profiles **$0.003 per comment** · Extract full comment threads, nested replies, and optional user profiles from **[Hacker News](https://news.ycombinator.com)** — the canonical tech discussion community. Uses the official **Firebase HN API**. **No key needed.** Built for **sentiment analysis on tech launches**, **YC / startup intelligence**, **community research**, **trend detection**, and **RAG/LLM corpora on developer discussions**. *** ### What You Get - **Full comment trees** — every reply in a story, with `parentId` linkage preserved - **Top stories scraping** — current HN front page with all their comments - **New stories scraping** — newest submissions with comments - **User profiles (opt-in)** — karma, about text, account age for each commenter - **Story metadata** — title, URL, points, author, timestamp alongside every comment - **Unlimited depth** — traverses the full nested thread, no arbitrary cutoffs - **Structured JSON** — comment-per-row, ready for analytics / LLM ingestion - **Public Firebase API** — no authentication, no quota headaches *** ### 4 Use Cases (ready-to-run JSON inputs) #### 1. Sentiment analysis on a specific launch ```json { "storyIds": [41009141], "maxCommentsPerStory": 0, "includeUserProfiles": false } ``` All comments under a single HN thread (0 = no cap). Feed into a sentiment/toxicity model to gauge reception of a product launch or announcement. #### 2. Daily "what's hot in tech" feed ```json { "scrapeTopStories": true, "maxStories": 10, "maxCommentsPerStory": 50, "includeUserProfiles": false } ``` Top 10 HN stories with 50 comments each — perfect for a morning tech digest or daily Slack summary. #### 3. Community intelligence — commenter profiles ```json { "storyUrls": ["https://news.ycombinator.com/item?id=41009141"], "maxCommentsPerStory": 200, "includeUserProfiles": true } ``` Comments + full profile (karma, bio, account age) for each commenter — great for mapping influential HN voices or finding experts in a niche. #### 4. Trend monitoring — new stories ```json { "scrapeNewStories": true, "maxStories": 30, "maxCommentsPerStory": 20, "includeUserProfiles": false } ``` 30 newest HN submissions with 20 comments each — useful for early-trend detection before stories hit the front page. *** ### Input Schema | Field | Type | Default | Description | |---|---|---|---| | `storyIds` | integer\[] | `[]` | HN story IDs (from URLs like `?id=41009141`) | | `storyUrls` | string\[] | `[]` | Full HN story URLs | | `scrapeTopStories` | boolean | `false` | Scrape current top stories | | `scrapeNewStories` | boolean | `false` | Scrape newest stories | | `maxStories` | integer | `10` | Max top/new stories to process | | `maxCommentsPerStory` | integer | `0` | Max comments per story (`0` = all) | | `includeUserProfiles` | boolean | `false` | Fetch commenter profile data | ### Output (sample — one comment row) ```json { "commentId": 41010231, "storyId": 41009141, "storyTitle": "Show HN: My new thing", "storyUrl": "https://news.ycombinator.com/item?id=41009141", "storyPoints": 312, "parentId": 41009141, "author": "pg", "text": "Congrats — this is great.

One suggestion...", "time": 1720123456, "timeISO": "2024-07-04T15:24:16Z", "depth": 1, "kids": [41010345, 41010512], "userProfile": { "karma": 155342, "about": "Founder of YC. Writes essays at paulgraham.com.", "created": 1160418633 } } ``` *** ### Pricing & Performance - **Pay-per-event:** $0.003 per comment (cheaper than $0.005 — HN comments are lightweight) - **Typical cost:** $0.03 for 10 comments, $0.30 for 100, $3 for 1,000 - **Speed:** ~50–80 comments/second (HN Firebase API is fast; 50 ms polite pacing) - **Free Apify tier:** $5/month credit = ~1,600 comments/month HN itself is free to browse — you pay only for structured extraction, thread assembly, and profile enrichment. *** ### Integrations - **Zapier / Make / n8n** — new comment on a keyword → Slack / Discord / Notion - **LangChain / LlamaIndex** — RAG over tech discussions and expert commentary - **Vector DBs (Pinecone / Weaviate / Qdrant)** — embed comments for semantic tech-discussion search - **Neo4j / Graphiti** — commenter → story → topic graph for community analytics - **Sentiment / toxicity APIs** — pipe `text` into Perspective API, OpenAI Moderations, or a local classifier - **Analytics DBs (BigQuery / Snowflake / DuckDB)** — HN comments as columnar tables for trend analysis - **Python SDK** ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("Helpermara/hackernews-comments-scraper").call( run_input={"scrapeTopStories": True, "maxStories": 5, "maxCommentsPerStory": 100, "includeUserProfiles": False} ) for c in client.dataset(run["defaultDatasetId"]).iterate_items(): print(c["storyTitle"], "—", c["author"], ":", c["text"][:80]) ``` *** ### FAQ **Do I need an HN API key?** No. The HN Firebase API (`hacker-news.firebaseio.com/v0`) is public and unauthenticated. **How fresh is the data?** Real-time. HN publishes updates to Firebase within seconds of user actions — you always get the current thread state. **How do I find a story ID?** From the URL: `news.ycombinator.com/item?id=41009141` → ID is `41009141`. Pass in `storyIds` or use `storyUrls` directly. **Does it handle deleted / dead comments?** Yes — the API flags them (`dead`, `deleted`). The actor returns them with the flag so you can filter downstream. **What about Ask HN / Show HN threads?** Those are regular HN stories — scrape them the same way. The `storyTitle` lets you filter by prefix (`Ask HN:`, `Show HN:`). **Differences vs the main `hackernews-scraper` actor?** This one focuses on **comments** (nested threads, profiles). The other focuses on **stories** (titles, URLs, points, ranking). Use together for full HN coverage. **Rate limits?** The Firebase API is generous and CDN-backed. The actor paces at 50 ms between requests to stay polite and fast. *** ### Keywords hacker news scraper, hn scraper, hn comments, hacker news api, hn threads, tech discussions, yc news, ycombinator scraper, startup intelligence, launch feedback, tech sentiment, developer community, hn users, hn karma, nested comments, firebase hn api, tech trends, show hn, ask hn, tech discussion mining *** ### Companions (cross-promo) - **[hacker-news-scraper](https://apify.com/Helpermara/hacker-news-scraper)** — HN stories (titles, URLs, points) - **[lobsters-scraper](https://apify.com/Helpermara/lobsters-scraper)** — Lobsters posts (tech discussion) - **[producthunt-scraper](https://apify.com/Helpermara/product-hunt-scraper)** — daily tech launches - **[devto-article-scraper](https://apify.com/Helpermara/devto-article-scraper)** — DEV.to articles *** ### Changelog - **2026-04-24** — Extended README with use cases, integrations, and FAQ - **2026-03** — Initial release: story IDs / URLs / top / new stories + optional user profiles # Actor input Schema ## `storyIds` (type: `array`): List of HN story IDs to scrape comments from (e.g. 41009141). Find IDs from HN URLs: news.ycombinator.com/item?id=XXXXX ## `storyUrls` (type: `array`): List of Hacker News URLs to scrape (e.g. https://news.ycombinator.com/item?id=41009141) ## `scrapeTopStories` (type: `boolean`): Scrape comments from current top stories on HN front page ## `scrapeNewStories` (type: `boolean`): Scrape comments from newest stories ## `maxStories` (type: `integer`): Max number of top/new stories to process ## `maxCommentsPerStory` (type: `integer`): Maximum number of comments to fetch per story (0 = all) ## `includeUserProfiles` (type: `boolean`): Fetch profile info for each commenter (karma, about, created) ## `flattenComments` (type: `boolean`): Output each comment as a separate dataset item (easier for analysis). If false, outputs nested tree structure. ## Actor input object example ```json { "scrapeTopStories": false, "scrapeNewStories": false, "maxStories": 10, "maxCommentsPerStory": 0, "includeUserProfiles": false, "flattenComments": true } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("openclawmara/hackernews-comments-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("openclawmara/hackernews-comments-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call openclawmara/hackernews-comments-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,openclawmara/hackernews-comments-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/1ylGzCGr8Z02E5VGH/builds/VKw9vhUwpYdxlYR1E/openapi.json