# Twitter/X Scraper: Tweets, Profiles & Search (`automation-lab/twitter-scraper`) Actor

Scrape X/Twitter tweets, user timelines, profiles, followers and following to JSON, CSV or Excel. Profiles and timelines need no login or API key; search and follower lists use your X cookies. Optional date filter for newest-first timelines. Pay only for the rows delivered.

- **URL**: https://apify.com/automation-lab/twitter-scraper.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** Social media, Lead generation
- **Features:** Limited permissions, Multi-tenant, Server interface
- **Stats:** 1,439 total users, 88 monthly users, 92.8% runs succeeded, 10 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $1.95 / 1,000 tweet scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Twitter/X Scraper: Tweets, Profiles & Search

Use this Twitter scraper / X scraper to collect profiles, tweets, user timelines, follower lists, following lists, and search results as JSON, CSV or Excel. Profiles and user timelines work without a login, search supports advanced X/Twitter query syntax, and direct HTTP requests keep runs lightweight compared with browser-based tweet scrapers.

### Why choose this X and tweet scraper?

| Need | What this actor gives you |
|------|---------------------------|
| One scraper for common Twitter/X jobs | Profiles, tweet URLs, user timelines, search, followers, and following in one actor |
| Start without account setup | Profiles and user timelines run with guest authentication — no cookies required |
| Advanced tweet discovery | Search accepts X/Twitter syntax such as `from:user`, `since:YYYY-MM-DD`, `min_faves:N`, `filter:media`, and `lang:en` |
| Clear auth expectations | Search, followers, and following require your Twitter cookies; tweets by URL may need cookies for restricted tweets |
| Lower runtime overhead | Direct HTTP implementation at 256 MB memory; no browser session is started for every page |

### What does Twitter/X Scraper do?

This actor extracts public data from Twitter/X:

- **Profiles** — follower counts, bio, join date, verification status, and more, without login
- **User timelines** — latest available tweets from any user's timeline with pagination, without login
- **Tweets by URL** — full tweet data from specific URLs including likes, retweets, media
- **Search** — find tweets by keyword, hashtag, or advanced query with full Twitter search syntax
- **Followers** — extract a user's follower list with profile details when cookies are provided
- **Following** — extract who a user follows with full profile data when cookies are provided

### Use this with a brand monitoring stack

Twitter/X is one layer of a complete brand-monitoring workflow on Apify. Pair Twitter/X Scraper with:

- [Reddit Scraper](https://apify.com/automation-lab/reddit-scraper) for community discussions, product feedback, and crisis signals
- [Threads Scraper](https://apify.com/automation-lab/threads-scraper) for public Threads posts and keyword searches
- [G2 Reviews & Products Scraper](https://apify.com/automation-lab/g2-scraper) for B2B review intelligence and competitor sentiment
- [Trustpilot Scraper](https://apify.com/automation-lab/trustpilot-scraper) for consumer review monitoring
- [Google News Scraper](https://apify.com/automation-lab/google-news-scraper) for media coverage and PR alerts
- [Website Change Monitor](https://apify.com/automation-lab/website-change-monitor) for competitor pricing, changelog, and landing-page changes

Together these actors form a custom brand-monitoring pipeline on Apify: social conversations, review sites, news coverage, and owned-web changes in one workflow.

### What works without login?

| Mode | Cookies needed? | Notes |
|------|----------------|-------|
| Profiles | No | Guest authentication, ~2 sec per profile |
| User timelines | No | Guest auth often returns a popular selection of about 100 tweets rather than a complete chronological feed |
| Tweets by URL | Recommended | Works without cookies for some tweets, but many require auth |
| Search | Yes | Native X search runs in your logged-in session. See [Native search reliability](#native-search-reliability) |
| Followers | Yes | Requires cookies — Twitter locks down follower lists |
| Following | Yes | Requires cookies — same as followers |

**Why do some modes need cookies?** Twitter has progressively locked down their API. Profiles and timelines still work with guest tokens, but native search, followers and following require an authenticated session, and many individual tweets do too. This actor uses direct HTTP requests with no browser. Other scrapers use different methods, so compare their requirements and cost per result for your own use case.

Getting cookies takes 30 seconds: open x.com in your browser, grab two values from DevTools, paste them in. Instructions below. Having both values in the input doesn't prove that X still accepts the session.

### Why use this scraper?

- **Tiered pay-per-event pricing** — starts at $0.003 per tweet on the FREE tier and discounts with paid Apify plans
- **No API keys needed** — profiles and timelines work without any login
- **No browser overhead** — pure HTTP at 256 MB memory
- **GraphQL endpoint lookup** — tries to read current GraphQL endpoint IDs from X's frontend JS on each run, with a built-in list if that lookup fails
- **Rich data** — likes, retweets, replies, views, bookmarks, quotes, media URLs, hashtags, mentions
- **Full search syntax** — from:user, since:date, min_faves:N, filter:media, lang:code, and more

### Who is it for?

#### Marketing and PR teams

- **Social media monitoring** — track brand mentions, competitor activity, or industry trends in real time
- **Campaign analysis** — measure tweet performance, hashtag reach, and engagement during campaigns
- **Crisis monitoring** — detect negative sentiment spikes and respond quickly

#### Sales and growth teams

- **Lead generation** — find potential customers through profile analysis and relevant conversations
- **Audience research** — understand what your target market talks about and who they follow

#### Research and analysis

- **Academic research** — collect tweets on specific topics for sentiment analysis, NLP, or social network studies
- **Content analysis** — study viral tweets, media usage, and hashtag trends
- **Influencer research** — analyze follower counts, engagement rates, and posting patterns

#### Developers and data teams

- **Archiving** — save specific tweets by URL for permanent record
- **Dataset building** — create structured Twitter datasets for ML training or analytics dashboards

### How much does it cost to scrape Twitter/X?

Pay-per-event pricing. You only pay for results you get.

For large runs, set **Maximum charge per run** in Apify before execution (especially with broad queries or `maxResults: 0`/very high values). This actor now checks the charge limit and exits when the PPE cap is reached.

| Event | Price | Notes |
|-------|-------|-------|
| Actor start | $0.005 FREE / $0.00475 BRONZE | Charged once, with the run's first result. Runs with no results pay nothing |
| Tweet scraped | $0.003 FREE / $0.0027 BRONZE | Per tweet (all modes) |
| Profile scraped | $0.003 FREE / $0.0027 BRONZE | Per profile |
| Follower/following scraped | $0.00115 FREE / $0.001 BRONZE | Per user in follower/following lists |

**Cost examples:**

| Task | Cost | Comparison |
|------|------|------------|
| 10 profiles | ~$0.035 | FREE tier |
| 100 tweets (user timeline) | ~$0.31 | FREE tier |
| 1,000 tweets (search) | ~$3.01 | FREE tier |
| 10,000 tweets | ~$30.01 | FREE tier |

The Apify free tier ($5/month) gets you roughly 1,600 tweets or 1,600 profiles per month before discounts or run-start charges.

### How to scrape Twitter/X

1. Open [Twitter/X Scraper](https://apify.com/automation-lab/twitter-scraper) in Apify Console
2. Pick a mode: Profiles, User timeline, Tweets by URL, or Search
3. Enter usernames, tweet URLs, or search queries
4. For search or tweets-by-URL: paste your Twitter cookies (see below)
5. Click **Start** and download results as JSON, CSV, or Excel

### How to get Twitter cookies

Required for search mode, recommended for tweets-by-URL.

1. Log in to [x.com](https://x.com) in your browser
2. Open DevTools (F12) → **Application** tab → **Cookies** → `https://x.com`
3. Find and copy the values of `auth_token` and `ct0`
4. Paste in the input field as: `auth_token=YOUR_TOKEN; ct0=YOUR_CT0`

Cookies typically last several weeks before you need to refresh them.

### What data can I extract from Twitter/X?

#### Tweet fields

| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Full tweet text (including long-form notes) |
| `url` | string | Permanent tweet URL |
| `likeCount` | number | Number of likes |
| `retweetCount` | number | Number of retweets |
| `replyCount` | number | Number of replies |
| `viewCount` | number | Number of views (timeline/search modes) |
| `bookmarkCount` | number | Number of bookmarks |
| `quoteCount` | number | Number of quote tweets |
| `createdAt` | string | Tweet creation date |
| `authorUsername` | string | Author's username |
| `authorFollowers` | number | Author's follower count |
| `hashtags` | array | Hashtags used |
| `mentions` | array | Mentioned users |
| `mediaUrls` | array | Photo and video URLs |
| `mediaType` | string | `photo`, `video`, `gif`, or `null` |
| `isReply` / `isRetweet` / `isQuote` | boolean | Tweet type flags |
| `inReplyToTweetId` / `inReplyToUsername` | string | Parent tweet and account for replies |
| `quotedTweet` | object | Summary of the quoted tweet (id, url, text, author) |
| `retweetedTweetId` / `retweetedUsername` | string | The original tweet for retweets; retweet rows carry the original's full text and counts |
| `conversationId` | string | Thread root tweet ID |

#### Profile fields

| Field | Type | Description |
|-------|------|-------------|
| `username` | string | Twitter handle |
| `name` | string | Display name |
| `bio` | string | Profile biography |
| `followers` | number | Follower count |
| `following` | number | Following count |
| `tweetsCount` | number | Total tweets posted |
| `likesCount` | number | Total likes given |
| `isBlueVerified` | boolean | Blue verification status |
| `joinedAt` | string | Account creation date |
| `profilePicture` | string | High-res profile image URL |
| `coverPicture` | string | Banner image URL |
| `website` | string | Website from profile |
| `location` | string | Location from profile |

### How do I scrape tweets and Twitter profiles?

| Parameter | Description | Default |
|-----------|-------------|---------|
| `mode` | `profiles`, `tweets`, `user-tweets`, `search`, `followers`, or `following` | `profiles` |
| `usernames` | Twitter usernames or profile links (for profiles, user-tweets, followers and following modes) | — |
| `tweetUrls` | Tweet URLs or numeric IDs (for tweets mode) | — |
| `searchTerms` | Search queries (for search mode) | — |
| `searchMode` | Sort order for search: `Top` or `Latest` | `Top` |
| `maxResults` | Max results per query/user | 50 |
| `sinceDate` | Only tweets on or after this date, for `user-tweets` (`2026-11-01` or `2026-11-01T12:00:00Z`). Older tweets are skipped and not charged. Needs a newest-first feed: for many large accounts X gives logged-out requests only an older popular selection, so recent tweets cannot be found and the status message says the filter could not be honoured. A run checks up to 20 accounts with nothing new, plus one more per delivered tweet; the status names any accounts left unchecked | — |
| `twitterCookie` | Twitter cookies (format: `auth_token=...; ct0=...`) | — |

#### Search operators

Search mode supports Twitter's full advanced search syntax:

| Operator | Example | Description |
|----------|---------|-------------|
| `from:user` | `from:elonmusk` | Tweets from a specific user |
| `to:user` | `to:NASA` | Tweets replying to a user |
| `since:date` | `since:2024-01-01` | Tweets after a date |
| `until:date` | `until:2024-12-31` | Tweets before a date |
| `min_faves:N` | `min_faves:100` | Minimum likes |
| `min_retweets:N` | `min_retweets:50` | Minimum retweets |
| `filter:media` | `web scraping filter:media` | Only tweets with media |
| `filter:images` | `apify filter:images` | Only tweets with images |
| `filter:videos` | `AI filter:videos` | Only tweets with videos |
| `lang:code` | `scraping lang:en` | Filter by language |
| `-filter:replies` | `from:NASA -filter:replies` | Exclude replies |

Combine operators freely: `from:elonmusk since:2024-01-01 min_faves:1000 lang:en`

### Output examples

#### Tweet

```json
{
    "type": "tweet",
    "id": "1519480761749016577",
    "url": "https://x.com/elonmusk/status/1519480761749016577",
    "text": "Next I'm buying Coca-Cola to put the cocaine back in",
    "createdAt": "Thu Apr 28 00:56:58 +0000 2022",
    "likeCount": 4253891,
    "retweetCount": 588370,
    "replyCount": 168937,
    "quoteCount": 167450,
    "bookmarkCount": 21524,
    "authorUsername": "elonmusk",
    "authorName": "Elon Musk",
    "authorFollowers": 236257999,
    "hashtags": [],
    "mentions": [],
    "mediaUrls": []
}
```

#### Profile

```json
{
    "type": "profile",
    "username": "elonmusk",
    "name": "Elon Musk",
    "followers": 236258375,
    "following": 1292,
    "tweetsCount": 98776,
    "likesCount": 215478,
    "isBlueVerified": true,
    "joinedAt": "Tue Jun 02 20:12:29 +0000 2009",
    "profilePicture": "https://pbs.twimg.com/profile_images/..._400x400.jpg"
}
```

#### Follower/Following

```json
{
    "type": "follower",
    "id": "44196397",
    "username": "elonmusk",
    "url": "https://x.com/elonmusk",
    "name": "Elon Musk",
    "bio": "",
    "location": "",
    "followers": 237387109,
    "following": 1301,
    "tweetsCount": 100007,
    "isVerified": false,
    "isBlueVerified": true,
    "joinedAt": "Tue Jun 02 20:12:29 +0000 2009",
    "profilePicture": "https://pbs.twimg.com/profile_images/..._400x400.jpg",
    "sourceUsername": "NASA",
    "scrapedAt": "2026-03-25T04:30:00.000Z"
}
```

### Rank collected tweets by bookmarks per like

Use `bookmarkCount / likeCount` to build a **bookmarks/likes shortlist**, not a statistical virality prediction or a comprehensive discovery feed. A high ratio on a tiny sample can be misleading; inspect the original links and absolute counts before drawing conclusions.

1. Collect tweet rows using `tweets`, `user-tweets`, or `search`, then export the dataset as **JSON** (an array). Search requires your own valid `twitterCookie`. Guest `user-tweets` returns popular posts, **not a chronological or complete feed**; ranking cannot recover posts that were never collected.
2. Keep only numeric, finite, nonnegative counts, with **bookmarks > 0 and likes > 0**. Zero likes means an undefined ratio: exclude it, never divide by zero or substitute one. Missing/null/string/negative/nonfinite metrics are unusable, not zero.
3. **This Actor normalizes absent upstream bookmark values to zero.** Stored zero cannot distinguish missing data from observed zero. Exclude every zero-bookmark row from this positive-signal shortlist; the example does not recover missingness or infer its provenance. Other normalized metrics can likewise lose upstream missingness.
4. Exclude invalid dates, future dates, and posts younger than 24 hours at a single recorded reference time. Exactly 24 hours old qualifies. Rank descending by bookmarks/likes, then bookmark count; ties use original URL in ascending code-point order, then input order.

#### Standalone public downstream example (Node.js 18+)

No packages, Apify SDK, credentials, network calls, plugins, or Actor changes are needed to run this example. Save the **entire** block as `rank-bookmarks.mjs`. It reads exported JSON from stdin and writes a local JSON shortlist. Supported dates are UTC ISO timestamps (with optional three-digit milliseconds) and the Actor's native Twitter date strings; other formats are rejected rather than guessed. Original tweet links are validated but never rewritten or fetched. This is downstream processing, not a new Actor input or output contract.

```javascript
import { readFileSync } from 'node:fs';
import assert from 'node:assert/strict';

function timestamp(value) {
    if (typeof value !== 'string') return NaN;
    const ms = Date.parse(value);
    if (!Number.isFinite(ms)) return NaN;
    const date = new Date(ms);
    const iso = date.toISOString();
    const native = date.toUTCString().replace(
        /^(\w+), (\d+) (\w+) (\d+) ([\d:]+) GMT$/,
        '$1 $3 $2 $5 +0000 $4',
    );
    return value === iso || value === iso.replace('.000Z', 'Z')
        || value === native ? ms : NaN;
}

function originalTweetLink(value) {
    if (typeof value !== 'string' || /\s/.test(value)) return false;
    try {
        const u = new URL(value);
        return u.protocol === 'https:' && !u.username && !u.password && !u.port
            && /^(www\.)?(x|twitter)\.com$/.test(u.hostname)
            && /^\/[A-Za-z0-9_]+\/status\/\d+(\/[^\s]*)?$/.test(u.pathname);
    } catch { return false; }
}

function rank(rows, referenceTime) {
    const now = timestamp(referenceTime);
    if (!Array.isArray(rows) || !Number.isFinite(now)) {
        throw new Error('Expected a JSON array and a valid UTC reference timestamp');
    }
    const usable = [];
    rows.forEach((row, index) => {
        if (!row || row.type !== 'tweet') return;
        const { bookmarkCount: bookmarks, likeCount: likes, createdAt, url } = row;
        if (![bookmarks, likes].every(n => typeof n === 'number'
            && Number.isFinite(n) && n >= 0) || bookmarks === 0 || likes === 0) return;
        const created = timestamp(createdAt);
        if (!Number.isFinite(created) || now - created < 24 * 60 * 60 * 1000) return;
        const bookmarksPerLike = bookmarks / likes;
        if (!Number.isFinite(bookmarksPerLike) || bookmarksPerLike <= 0
            || !originalTweetLink(url)) return;
        usable.push({ url, createdAt, bookmarkCount: bookmarks,
            likeCount: likes, bookmarksPerLike, index });
    });
    usable.sort((a, b) => b.bookmarksPerLike - a.bookmarksPerLike
        || b.bookmarkCount - a.bookmarkCount
        || (a.url < b.url ? -1 : a.url > b.url ? 1 : 0) || a.index - b.index);
    return usable.map(({ index, ...row }) => row);
}

if (process.argv[2] === '--self-test') {
    // Synthetic offline fixtures, not observed posts or performance evidence.
    const now = '2026-10-20T12:00:00Z';
    const base = { type: 'tweet', url: 'https://x.com/example/status/1',
        createdAt: '2026-10-19T12:00:00Z', bookmarkCount: 2, likeCount: 10 };
    const result = (patch) => rank([{ ...base, ...patch }], now);
    assert.equal(result({}).length, 1); // Exact 24h boundary.
    assert.equal(result({ createdAt: 'Mon Oct 19 12:00:00 +0000 2026' }).length, 1);
    for (const key of ['bookmarkCount', 'likeCount']) {
        for (const value of [undefined, null, '2', -1, NaN, Infinity, -Infinity, 0]) {
            assert.equal(result({ [key]: value }).length, 0);
        }
    }
    for (const createdAt of [null, '', 'invalid', '2026-02-30T12:00:00Z',
        '2026-10-21T12:00:00Z', '2026-10-19T12:00:00.001Z']) {
        assert.equal(result({ createdAt }).length, 0);
    }
    assert.equal(result({ bookmarkCount: 1.5, likeCount: 3 })[0].bookmarksPerLike, 0.5);
    assert.equal(result({ bookmarkCount: Number.MAX_VALUE, likeCount: Number.MIN_VALUE }).length, 0);
    assert.equal(result({ type: 'profile' }).length, 0);
    assert.equal(result({ url: 'https://example.invalid/status/1' }).length, 0);
    const original = 'https://twitter.com/example/status/1?ref=fixture#original';
    assert.equal(result({ url: original })[0].url, original);
    const rows = [base, { ...base, url: 'https://x.com/example/status/2' },
        { ...base, bookmarkCount: 4, likeCount: 20 },
        { ...base, bookmarkCount: 3, likeCount: 10 }];
    const ordered = rank(rows, now);
    assert.deepEqual(ordered.map(r => r.bookmarkCount), [3, 4, 2, 2]);
    assert.equal(ordered[2].url, base.url);
    assert.deepEqual(rank([...rows].reverse(), now), ordered);
    const duplicates = [base, { ...base, createdAt: '2026-10-18T12:00:00Z' }];
    assert.deepEqual(rank(duplicates, now).map(r => r.createdAt), duplicates.map(r => r.createdAt));
    assert.deepEqual(rank([null, {}, base], now), rank([base], now));
    assert.throws(() => rank({}, now));
    assert.throws(() => rank([], 'invalid'));
    console.log('Offline bookmark-ranking fixtures passed');
} else {
    const referenceTime = process.argv[2] || new Date().toISOString();
    console.log(JSON.stringify({ referenceTime, metric: 'bookmarks/likes',
        shortlist: rank(JSON.parse(readFileSync(0, 'utf8')), referenceTime) }, null, 2));
}
```

Run the embedded fixtures, then process your downloaded export (no live scraping is performed by these commands):

```bash
node rank-bookmarks.mjs --self-test
node rank-bookmarks.mjs 2026-10-20T12:00:00Z < tweets.json > shortlist.json
## Omit the timestamp to use current UTC time; retain referenceTime with the report.
```

For example, a synthetic tweet with `bookmarkCount: 2`, `likeCount: 10`, and a valid link/date at least 24 hours old yields `bookmarksPerLike: 0.2`. That means 0.2 bookmarks per like, not a 20% chance of going viral. An empty shortlist means no collected rows met these rules, not that no relevant posts exist. Input order is retained only for otherwise identical ranking keys; duplicates are not silently removed.

#### Customer-owned automation

You can schedule your own configured Actor/task in Apify Console, retain each successful run's dataset ID, export JSON, and run this script in your own downstream job. Do not publish cookies or put them in prompts; keep account secrets in your customer-owned configuration. Scheduling, storing exports, and any later notification are your responsibility; this recipe does not install anything or send messages.

For a compatible, customer-authorized MCP client, use `https://mcp.apify.com?tools=automation-lab/twitter-scraper`. Discover actual tool names and argument schemas with scoped `tools/list`; this selection includes run/data helpers as well as the Actor, not exactly one tool. Map `mode`, `usernames`/`tweetUrls`/`searchTerms`, and `maxResults` to the same Actor inputs. Start once, retain the run ID/status/storage IDs, then follow **that same run** using `get-actor-run`. `call-actor` waits at most 45 seconds (`waitSecs`, default 30); use client timeout above that wait plus transport margin. Example consumer policy: back off 2, 4, 8 seconds capped at 10, stopping at 120 seconds total. On timeout recover the known run rather than starting another; if the run ID is unknown, report uncertainty. At the deadline report pending and stop. Terminal failures are failures, and any rows are partial.

After success, use `get-dataset-items` with the returned `defaultDatasetId`, explicit `offset`, `limit`, and fields `type,url,createdAt,bookmarkCount,likeCount` as supported by the discovered schema. Process untransformed JSON host-side using this example. Example model-context budget: pages of 20 source rows, at most 100 rows and 64 KiB serialized UTF-8 total; track source offsets, not shortlist length. Enforce bytes before injecting content, including oversized pages/rows; otherwise do not promise a hard byte bound. Report scanned/returned counts, continuation offset, and partial/truncated status when limited. Keep full exports outside model context; these consumer limits do not change Actor collection defaults. Client discovery/caching is client-dependent, not a universal context-savings guarantee. See [official MCP guidance](https://docs.apify.com/platform/integrations/mcp). This documents an existing integration, **not a guaranteed Grok plugin or installation path**.

Keep Instagram analysis in a separate report: public comments/likes, if supplied by a lawful separate source, are a **comments/likes proxy**, not public saves, bookmarks, or an equivalent cross-platform score. This Actor does not collect Instagram data or expose Instagram saves.

### How do I get the best results from Twitter Scraper?

- **Start small** — test with `maxResults: 10` before scaling up
- **User timeline order** — guest auth returns popular tweets, not chronological. Use search mode with `from:username` + cookies for chronological order
- **Endpoint lookup** — the actor tries to read GraphQL endpoint IDs from X's frontend JS on each run. Some X changes can still need an actor update
- **Rate limits** — Twitter throttles requests; the actor adds delays automatically, so large runs may take longer
- **Combine modes** — use Search to discover relevant accounts, then User Timeline to collect available timeline posts from specific users
- **Use advanced search operators** — narrow results with `min_faves:100`, `filter:media`, or `lang:en` to get higher-signal data and reduce costs
- **Schedule recurring runs** — set up daily or weekly scrapes in Apify Console to monitor accounts or keywords over time automatically

### How to scrape Twitter without getting blocked

Twitter actively limits access to its data. Here's how this scraper handles that, and where the limits are:

- **Direct HTTP, not browser automation** — the actor sends raw HTTP requests, modelled on what the X frontend does. It runs no browser and uses 256 MB of memory.
- **GraphQL endpoint lookup** — Twitter periodically rotates its internal GraphQL endpoint IDs. This scraper tries to read the current IDs from X's frontend JavaScript on every run. If that lookup fails, it uses a built-in list that can be out of date. This is designed to follow ID rotations, but other X changes can still need an actor update.
- **Guest token authentication** — profiles and user timelines use guest tokens (no login required), which are less aggressively rate-limited than API keys.
- **Clean stop at the run timeout** — about 20 seconds before the timeout the Actor stops, keeps every row saved so far, and reports in the status message how many inputs were not finished. Each profile takes about 4 seconds, so the default 600-second timeout covers roughly 150 usernames; raise the timeout in Run options or split larger lists. If nothing was saved in time, the run fails.
- **Bounded transient-failure recovery** — rate limits, temporary 5xx responses, socket resets, timeouts, and TLS disconnects get up to three attempts in total, with exponential backoff and jitter. Authentication, validation, and not-found failures still return immediately.
- **Authenticated sessions for search** — search requires browser cookies (`auth_token` + `ct0`) from a session X still accepts. Cookies typically last several weeks. If you get auth errors, refresh them from x.com in under a minute.

For large-scale runs (10K+ tweets), spread queries across multiple runs and use the `since:` / `until:` operators to batch by date range.

#### Native search reliability

Native keyword search has had intermittent connection failures, including with **Top** sort order. In the observed case, requests timed out or ended in proxy 590 and HTTP/2 `SETTINGS` errors, and X returned no response. This doesn't mean every authenticated search fails.

- **Current mitigation:** each search retry now moves to a fresh residential connection, within the same three attempts, and those HTTP/2 failures are now retried. If X returns errors and no search results, the run fails with a clear error. A search that really has no matches still succeeds with an empty dataset.
- **What we've verified:** we tested the retry mechanics, charges, privacy handling and result delivery on the current build.
- **What we haven't verified:** we haven't confirmed useful results from a real logged-in search on this build, or that this mitigation resolves the failure. Treat it as a mitigation, not a guaranteed fix.
- **Cookies:** cookies are required, but having them in the input doesn't prove they're valid. Bad cookies aren't an established cause of these connection failures, because X sent no response to judge them by. Authentication errors point to your cookies, so refreshing them may help there, but it may not help with a timeout or connection error.
- **Search source:** results come only from native X search. The actor doesn't switch to another search source when native search fails.

### Twitter data export: how to download tweets

Once a run completes, you can download your scraped tweets in multiple formats:

- **JSON** — full structured data, best for developers and data pipelines
- **CSV** — opens directly in Excel or Google Sheets, ideal for analysts
- **Excel (.xlsx)** — pre-formatted spreadsheet, ready to share
- **XML** — compatible with legacy enterprise systems

**Via Apify Console:** Click **Export** on the dataset page and choose your format.

**Via API:** Use the dataset export endpoint:

```bash
curl "https://api.apify.com/v2/datasets/DATASET_ID/items?format=csv&token=YOUR_TOKEN" \
  --output tweets.csv
```

**Automatic export integrations:** Connect the scraper to Google Sheets, Airtable, or a webhook endpoint so data is exported automatically after each run — no manual downloads needed.

### API usage

#### Node.js

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('automation-lab/twitter-scraper').call({
    mode: 'profiles',
    usernames: ['elonmusk', 'OpenAI'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("automation-lab/twitter-scraper").call(run_input={
    "mode": "user-tweets",
    "usernames": ["NASA"],
    "maxResults": 50,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

#### cURL

```bash
curl "https://api.apify.com/v2/acts/automation-lab~twitter-scraper/runs" \
  -X POST \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode": "profiles", "usernames": ["elonmusk"]}'
```

### Integrations

Connect Twitter/X Scraper to your workflow using [Apify integrations](https://apify.com/integrations):

- **Google Sheets** — export tweets and profiles to a spreadsheet automatically. Useful for tracking follower growth or building engagement reports
- **Slack / Discord** — get notifications when scraping finishes or when tweets match specific criteria
- **Zapier / Make** — automate workflows triggered by scraping results, e.g., save viral tweets to Airtable or trigger alerts for brand mentions
- **Webhooks** — send data to your API endpoint automatically for custom processing
- **Scheduled runs** — set up recurring scrapes (hourly, daily, weekly) to monitor accounts or search queries over time
- **Data warehouses** — pipe results to BigQuery, Snowflake, or PostgreSQL for large-scale analytics
- **AI/LLM pipelines** — feed tweets into sentiment analysis, topic classification, or trend detection models

### Use with AI agents via MCP

Twitter/X Scraper is available as a tool for AI assistants that support the [Model Context Protocol (MCP)](https://docs.apify.com/platform/integrations/mcp). This lets you use natural language to scrape data — just ask your AI assistant and it will configure and run the scraper for you.

#### Setup for Claude Code

```bash
claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/twitter-scraper"
```

#### Setup for Claude Desktop, Cursor, or VS Code

Add this to your MCP config file:

```json
{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=automation-lab/twitter-scraper"
        }
    }
}
```

Your AI assistant will use OAuth to authenticate with your Apify account on first use.

#### Example prompts

Once connected, try asking your AI assistant:

- "Get the latest 200 tweets from @elonmusk"
- "Search Twitter for tweets about 'climate change' from the past week"
- "Get the first 500 followers of @NASA with their profile details"

Learn more in the [Apify MCP documentation](https://docs.apify.com/platform/integrations/mcp).

### Legality

This actor scrapes publicly available data from Twitter/X. It does not access private accounts, direct messages, or any data behind login walls.

Web scraping of public data is generally legal in most jurisdictions. Always respect Twitter's Terms of Service and applicable data protection laws (GDPR, CCPA) when using scraped data.

This actor is independently operated and is not affiliated with, endorsed by, or sponsored by X Corp. or Twitter. X and Twitter are used only to identify the public data source.

### Data handling and retention

The actor sends source requests only to public X/Twitter endpoints through the configured Apify proxy. It does not call AI providers or keep a separate external copy of successful results. Failed operations send sanitized diagnostic input, exceptions, and Actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL queries are removed; diagnostic reports are retained for 30 days. Cookie input is treated as a secret and is not written to actor logs or dataset rows. Run input, datasets, key-value stores, and logs remain in the user's Apify account and follow that account's storage and deletion settings; users can delete them with Apify's storage controls.

### FAQ

**Does this need a Twitter API key?**
No. Profiles and user timelines use guest authentication. Search and tweets-by-URL use your browser cookies (not API keys).

**Why do some modes need cookies?**
Twitter has locked down their API over the past year. Guest tokens still work for profiles and timelines, but search and many individual tweet lookups require authenticated sessions. We use direct HTTP with no browser.

**Why did my search fail with a timeout or connection error?**
Native search has had intermittent connection failures that don't come with a response from X. Retries now use a fresh connection, but we haven't confirmed that this resolves the failure. See [Native search reliability](#native-search-reliability).

**Can I scrape private accounts?**
No. Only publicly available data.

**How many tweets can I scrape?**
Up to 5,000 per user in user-tweets mode. Search mode supports pagination up to 5,000 results per query.

**What if Twitter rotates their API?**
The actor tries to read endpoint IDs from X's frontend JavaScript on each run and uses a built-in list if that fails. This is designed to follow ID rotations, but other API changes can still need an actor update.

**Why are some tweets-by-URL empty?**
Some tweets return empty data even with valid cookies. This happens with restricted, age-gated, or visibility-limited tweets. This is a Twitter API limitation.

**Can I scrape a user's follower list?**
Yes. Use `followers` or `following` mode with the target username. Set `maxResults` to control how many to extract. Both modes require cookies. Large accounts (1M+ followers) may take longer due to Twitter throttling, and availability depends on your authenticated session.

**How long do cookies last?**
Typically several weeks. If you get authentication errors, refresh your cookies from x.com.

**The scraper returns 0 results for user-tweets mode — what's wrong?**
Guest authentication for user-tweets mode returns popular tweets, not all tweets. If the user hasn't posted recently popular content, the result set may be small. Try search mode with `from:username` and cookies for more comprehensive results.

**I'm getting "Could not authenticate" errors — how do I fix this?**
Your cookies have likely expired. Log in to x.com in your browser again, open DevTools (F12) > Application > Cookies > x.com, and copy fresh values for `auth_token` and `ct0`. Paste them in the format: `auth_token=YOUR_TOKEN; ct0=YOUR_CT0`.

### Related scrapers

- [Threads Scraper](https://apify.com/automation-lab/threads-scraper) — Extract posts and profiles from Meta's Threads
- [TikTok Scraper](https://apify.com/automation-lab/tiktok-scraper) — Scrape TikTok videos, profiles, and trending hashtag feeds
- [Reddit Scraper](https://apify.com/automation-lab/reddit-scraper) — Scrape posts, comments, and communities from Reddit
- [Bluesky Scraper](https://apify.com/automation-lab/bluesky-scraper) — Scrape Bluesky posts, profiles, and search results
- [Pinterest Scraper](https://apify.com/automation-lab/pinterest-scraper) — Scrape Pinterest pins, boards, and profile data

### Use with AI agents via x402 (no Apify account)

Agent workflows can run this actor through Apify MCP without an Apify account when the MCP client uses a funded x402 wallet. Use the actor-specific MCP endpoint:

`https://mcp.apify.com?tools=automation-lab/twitter-scraper&payment=x402`

This keeps existing Apify MCP/API usage unchanged; x402 only changes payment/authentication for clients that support it. Learn more in Apify's [x402 integration docs](https://docs.apify.com/platform/integrations/x402).

### Server API: get X profiles and tweets over HTTP

Twitter/X Scraper is also a Server Actor. Besides normal runs, it answers short HTTP requests directly, without starting a run for each lookup. Copy the server URL from the Actor's **Server** tab (do not build it from the Actor name) and authenticate with your Apify token. The server uses only no-login access: it never takes X cookies. Use normal Actor runs for many accounts, more than 40 tweets, `sinceDate`, search, followers and following.

```bash
curl -sS "$SERVER_URL/profile?username=NASA" -H "Authorization: Bearer $APIFY_TOKEN"
curl -sS "$SERVER_URL/tweets?username=NASA&maxTweets=10" -H "Authorization: Bearer $APIFY_TOKEN"
curl -sS "$SERVER_URL/tweet?id=https://x.com/NASA/status/1585341984679469056" -H "Authorization: Bearer $APIFY_TOKEN"
```

| Method and path | Input | Output and charge |
| --- | --- | --- |
| `GET /` | None | `ready: true`, the routes and limits; no scraping or charge |
| `GET /profile` | Query `username`: an X handle such as `NASA`, `@NASA` or a profile link such as `https://x.com/NASA` | `{ok,count,results}` with the same record as a normal run's `profiles` mode item; one Profile event; no start fee |
| `GET /tweets` | Query `username` as above; optional `maxTweets` 1–40 (default 20) | `{ok,count,results}` with up to 40 tweets from the first page of the account's timeline, the same records as a normal run's `user-tweets` mode items; one Tweet event per returned tweet; no start fee |
| `GET /tweet` | Query `id`: a tweet ID or a tweet URL such as `https://x.com/NASA/status/1585341984679469056` | `{ok,count,results}` with the same record as a normal run's `tweets` mode item without cookies; one Tweet event; no start fee |

X decides which tweets a logged-out timeline shows; for some large accounts it is a popular selection rather than the newest. An unknown account, an empty timeline or a tweet X does not show logged-out returns an error and is free. Each request has a 60-second deadline and a 1 MiB response cap, and one server works on at most eight requests at once. Nothing is cached. Errors use `{ "ok": false, "error": { "code": "...", "message": "...", "retryable": false } }`: 400 `INVALID_USERNAME`, `INVALID_LIMIT`, `INVALID_TWEET` or `INVALID_INPUT`; 402 spending limit; 404 `USER_NOT_FOUND`, `NO_RESULTS`, `TWEET_NOT_FOUND` or unknown route; 405 wrong method; 413 `RESPONSE_TOO_LARGE`; 502 `BLOCKED`, `UPSTREAM_ERROR` or charging failure; 503 server busy or stopping; 504 `TIMEOUT`. Failed requests are free. Retry only when `retryable` is true. The Server tab shows the full OpenAPI description.

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/twitter-scraper/changelog.md

# Actor input Schema

## `mode` (type: `string`):

Select what to scrape from Twitter/X.

## `usernames` (type: `array`):

Enter Twitter/X usernames or profile links for profile, user-tweets, followers, or following modes. Examples: NASA, @NASA or https://x.com/NASA.

## `tweetUrls` (type: `array`):

Enter tweet URLs (https://x.com/user/status/123 or https://twitter.com/user/status/123) or numeric tweet IDs.

## `searchTerms` (type: `array`):

Enter search terms or Twitter advanced search queries. Supports operators like from:user, since:2024-01-01, min_faves:100, filter:media, lang:en.

## `searchMode` (type: `string`):

Sort search results by relevance (Top) or chronologically (Latest).

## `sinceDate` (type: `string`):

Optional date for user-tweets mode, such as 2026-11-01 or 2026-11-01T12:00:00Z. Tweets older than this are skipped and not charged. Works when X serves the timeline newest-first; for many large accounts X gives logged-out requests only an older popular selection, so recent tweets cannot be found and the status message says so. A run checks up to 20 accounts with nothing new, plus one more per delivered tweet, and the status names any accounts left unchecked. For search mode use the since: operator in the query instead.

## `maxResults` (type: `integer`):

Maximum number of results to return per query/username in user-tweets, search, followers, and following modes.

## `twitterCookie` (type: `string`):

Required for search, followers, and following modes; recommended for tweets-by-URL. Paste cookies in this format: auth_token=YOUR_TOKEN; ct0=YOUR_CT0. Get them from browser DevTools → Application → Cookies → x.com.

## Actor input object example

```json
{
  "mode": "user-tweets",
  "usernames": [
    "NASA"
  ],
  "tweetUrls": [
    "https://x.com/apify/status/2102050719192813987"
  ],
  "searchTerms": [
    "apify web scraping"
  ],
  "searchMode": "Top",
  "maxResults": 20,
  "twitterCookie": "auth_token=YOUR_AUTH_TOKEN; ct0=YOUR_CT0"
}
```

# Actor output Schema

## `overview` (type: `string`):

Scraped Twitter/X records in the overview dataset view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "user-tweets",
    "usernames": [
        "NASA"
    ],
    "tweetUrls": [
        "https://x.com/apify/status/2102050719192813987"
    ],
    "searchTerms": [
        "apify web scraping"
    ],
    "maxResults": 20,
    "twitterCookie": "auth_token=YOUR_AUTH_TOKEN; ct0=YOUR_CT0"
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/twitter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "user-tweets",
    "usernames": ["NASA"],
    "tweetUrls": ["https://x.com/apify/status/2102050719192813987"],
    "searchTerms": ["apify web scraping"],
    "maxResults": 20,
    "twitterCookie": "auth_token=YOUR_AUTH_TOKEN; ct0=YOUR_CT0",
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/twitter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "user-tweets",
  "usernames": [
    "NASA"
  ],
  "tweetUrls": [
    "https://x.com/apify/status/2102050719192813987"
  ],
  "searchTerms": [
    "apify web scraping"
  ],
  "maxResults": 20,
  "twitterCookie": "auth_token=YOUR_AUTH_TOKEN; ct0=YOUR_CT0"
}' |
apify call automation-lab/twitter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/twitter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tIpjRQruPOB39HLf5/builds/pOZ26ewp4x5cd5slM/openapi.json