# Hacker News Scraper — Stories, Comments & Jobs (`cryptosignals/hackernews-scraper`) Actor

Scrape Hacker News stories, comments, and user profiles — extract title, URL, score, author, comment threads, and submission time. CSV/JSON output.

- **URL**: https://apify.com/cryptosignals/hackernews-scraper.md
- **Developed by:** [Web Data Labs](https://apify.com/cryptosignals) (community)
- **Categories:** Social media, News, AI
- **Features:** Limited permissions
- **Stats:** 8 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$5.00 / 1,000 result scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper: Stories, Comments & Users Free

The most comprehensive **Hacker News scraper** on Apify. Extract HN stories, comments, and user profiles at scale. Search Hacker News posts by keyword. Export to JSON, CSV, or Excel. Built on the official HN Firebase API for maximum reliability.

### Why Use This Hacker News Scraper?

- **Tech Trend Analysis** - Track what's trending on Hacker News. Monitor emerging technologies, frameworks, and tools before they go mainstream.
- **Startup Scouting** - Discover new startups from Show HN and Launch HN posts. Get early intelligence on companies before they hit the press.
- **Hiring Intelligence** - Monitor "Who is Hiring" threads and job postings. Analyze which companies are hiring and for what roles.
- **Competitive Research** - Track mentions of competitors, products, or technologies. See what the developer community thinks.
- **Content Curation** - Build datasets of top-performing HN content. Understand what resonates with the tech community.
- **Academic Research** - Analyze HN data for sentiment analysis, trend detection, and community dynamics research.

### Features

| Feature | Description |
|---------|-------------|
| **6 Story Categories** | Top, New, Best, Ask HN, Show HN, Jobs |
| **Full-Text Search** | Search HN posts via Algolia API |
| **Comment Trees** | Fetch full comment threads for any story |
| **User Profiles** | Get karma, about, and submission counts |
| **Pagination** | Control output size with maxItems |
| **Concurrent Fetching** | Fast parallel requests with rate limiting |
| **Multiple Export Formats** | JSON, CSV, Excel, XML via Apify |

### Input Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `category` | string | `"top"` | Story category: `top`, `new`, `best`, `ask`, `show`, `jobs`, or `search` |
| `searchQuery` | string | `""` | Search query (required when category is `search`) |
| `maxItems` | integer | `100` | Maximum stories to return (1-500) |
| `includeComments` | boolean | `false` | Fetch comment trees for each story |
| `maxCommentsPerStory` | integer | `50` | Max comments per story |
| `scrapeType` | string | `"stories"` | Output type: `stories`, `users`, or `both` |

### Output - Stories

```json
{
  "id": 12345678,
  "title": "Show HN: I built an AI-powered code reviewer",
  "url": "https://example.com/project",
  "text": null,
  "author": "techfounder",
  "score": 342,
  "commentCount": 156,
  "createdAt": "2025-12-15T10:30:00.000Z",
  "storyType": "show",
  "hnUrl": "https://news.ycombinator.com/item?id=12345678"
}
```

### Output - Users

```json
{
  "username": "techfounder",
  "karma": 15420,
  "about": "Building cool stuff. Previously at BigCorp.",
  "createdAt": "2015-03-10T08:00:00.000Z",
  "submittedCount": 892
}
```

### Output - Comments (when includeComments is true)

```json
{
  "id": 12345679,
  "author": "commenter1",
  "text": "This is a great project! I've been looking for something like this.",
  "createdAt": "2025-12-15T11:00:00.000Z",
  "parentId": 12345678
}
```

### Example Use Cases

#### Get Top 50 HN Stories

```json
{
  "category": "top",
  "maxItems": 50
}
```

#### Search for AI Startup Posts

```json
{
  "category": "search",
  "searchQuery": "AI startup launch",
  "maxItems": 100
}
```

#### Get Ask HN with Comments

```json
{
  "category": "ask",
  "maxItems": 20,
  "includeComments": true,
  "maxCommentsPerStory": 100
}
```

#### Get Stories + Author Profiles

```json
{
  "category": "best",
  "maxItems": 30,
  "scrapeType": "both"
}
```

### API Usage Examples

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('cryptosignals/hackernews-scraper').call({
    category: 'top',
    maxItems: 50,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("cryptosignals/hackernews-scraper").call(run_input={
    "category": "top",
    "maxItems": 50,
})
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(items)
```

#### cURL

```bash
curl "https://api.apify.com/v2/acts/cryptosignals~hackernews-scraper/runs" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -d '{"category": "top", "maxItems": 50}'
```

### About the Data Source

This scraper uses the **official Hacker News Firebase API** (https://github.com/HackerNews/API) and the **Algolia HN Search API** for keyword search. These are public, free APIs provided by Y Combinator. No scraping of the website is involved -- all data comes from official endpoints, ensuring reliability and compliance.

### FAQ

**Q: Is this scraper free to use?**
A: The actor itself is free. You only pay for Apify compute units based on your plan.

**Q: How often is the data updated?**
A: The HN API provides real-time data. Stories and scores are current at the time of scraping.

**Q: Can I scrape historical data?**
A: Use the search feature to find older posts by keyword. The Firebase API only provides current story lists.

**Q: Why are some comments missing?**
A: Deleted or dead (flagged) comments are filtered out automatically.

**Q: What's the rate limit?**
A: The official HN API has no documented rate limit. The scraper uses concurrent requests with sensible batching.

**Q: Can I schedule this to run automatically?**
A: Yes! Use Apify's scheduling feature to run the scraper hourly, daily, or on any custom schedule.

### Tags

hacker news, hn scraper, tech news, startup intelligence, developer tools, comment scraper, user profiles, tech trends, Y Combinator, hackernews api, hn data extraction

***

*Found this actor useful? [Leave a review](https://apify.com/cryptosignals/hackernews-scraper#reviews) - it helps a lot. Something broken? Report it on the [Issues tab](https://apify.com/cryptosignals/hackernews-scraper/issues).*

# Actor input Schema

## `category` (type: `string`):

Which HN category to scrape. Use 'search' to search by keyword.

## `searchQuery` (type: `string`):

Search query (only used when category is 'search'). Searches HN posts via Algolia.

## `maxItems` (type: `integer`):

Maximum number of stories/users to return.

## `includeComments` (type: `boolean`):

Fetch the full comment tree for each story. Warning: significantly increases run time.

## `maxCommentsPerStory` (type: `integer`):

Maximum number of comments to fetch per story (when includeComments is true).

## `scrapeType` (type: `string`):

What to scrape: stories only, user profiles of story authors, or both.

## Actor input object example

```json
{
  "category": "top",
  "maxItems": 100,
  "includeComments": false,
  "maxCommentsPerStory": 50,
  "scrapeType": "stories"
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("cryptosignals/hackernews-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("cryptosignals/hackernews-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call cryptosignals/hackernews-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cryptosignals/hackernews-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0UDODOnpTkxY3Oc90/builds/aVcgeM6Oi2My63Isl/openapi.json