# X Profile Scraper (`devwithbobby/x-profile-scraper`) Actor

X Profile scraper

- **URL**: https://apify.com/devwithbobby/x-profile-scraper.md
- **Developed by:** [Dev with Bobby](https://apify.com/devwithbobby) (community)
- **Categories:** Automation, Lead generation, Social media
- **Features:** Limited permissions
- **Stats:** 4 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## X (Twitter) Profile Scraper

Extract X (Twitter) profile data with advanced anti-detection measures. Provide your session cookies, enter usernames or URLs, and get structured data instantly.

> **Important:** X/Twitter requires login to view profiles. You must provide your session cookies for this scraper to work.

### Features

- **Simple Input** - Accepts usernames, @handles, full URLs, or mixed formats
- **Profile Data** - Display name, bio, location, website, join date, followers, following
- **Engagement Metrics** - Followers count, following count, tweets count, likes count
- **Verification Status** - Detects blue verified and legacy verified badges
- **Media** - Profile picture (high-res) and banner image URLs
- **Anti-Detection** - Browser fingerprinting, stealth headers, popup dismissal
- **Graceful Degradation** - Returns partial data when full access is blocked

### Prerequisites: Getting Your Cookies

X/Twitter now requires authentication to view any profile data. You must provide your session cookies.

#### Step-by-Step Guide

1. **Install Cookie-Editor extension**
   - [Chrome](https://chrome.google.com/webstore/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)
   - [Firefox](https://addons.mozilla.org/en-US/firefox/addon/cookie-editor/)

2. **Log into X/Twitter**
   - Go to [x.com](https://x.com) and log in with your account

3. **Export your cookies**
   - Click the Cookie-Editor extension icon
   - Click **"Export"** button
   - Select **"Export as JSON"**
   - Copy the entire JSON array

4. **Paste into the actor**
   - Paste the JSON into the `cookies` input field

#### Required Cookies

The scraper validates that you have these essential cookies:

| Cookie | Description |
|--------|-------------|
| `auth_token` | Your session authentication token (required) |
| `ct0` | CSRF token for API requests (recommended) |

#### Example Cookie Format

```json
[
  {"name": "auth_token", "value": "abc123def456...", "domain": ".x.com", "path": "/", ...},
  {"name": "ct0", "value": "xyz789...", "domain": ".x.com", "path": "/", ...},
  ...
]
```

> **Note:** Cookie-Editor exports all cookies with the correct format. Just paste the entire output.

### Usage

#### Input Formats

Enter X/Twitter profiles in any format (one per line or comma-separated):

```
elonmusk
@OpenAI
https://x.com/BillGates
https://twitter.com/satloops
x.com/apaborkar
```

#### Input Parameters

| Parameter | Type | Description |
|-----------|------|-------------|
| `profiles` | String | X/Twitter usernames or URLs (required) |
| `cookies` | String | X session cookies as JSON array (required) |
| `proxyType` | String | Proxy group: `RESIDENTIAL` (default), `BUYPROXIES94952` (Datacenter), or `StaticUS3` |
| `maxConcurrency` | Integer | Parallel pages (1-3, default: 1) |
| `maxRequestsPerMinute` | Integer | Rate limit (5-20, default: 10) |

### Output

#### Successful Scrape

```json
{
  "inputUrl": "https://x.com/elonmusk",
  "scrapedUrl": "https://x.com/elonmusk",
  "username": "elonmusk",
  "displayName": "Elon Musk",
  "bio": "The people voted for major change...",
  "location": "Earth",
  "website": "https://x.com/America",
  "joinedDate": "June 2009",
  "followers": 215000000,
  "following": 1024,
  "tweetsCount": 65000,
  "likesCount": 52000,
  "isVerified": false,
  "isBlueVerified": true,
  "profilePicture": "https://pbs.twimg.com/profile_images/.../photo_400x400.jpg",
  "bannerImage": "https://pbs.twimg.com/profile_banners/.../1500x500",
  "recentPosts": [
    {
      "id": "1883847583920193820",
      "text": "This is the tweet content...",
      "timestamp": "2026-01-26T08:30:00.000Z",
      "timeAgo": "2h",
      "url": "https://x.com/elonmusk/status/1883847583920193820",
      "replies": 15420,
      "reposts": 32100,
      "likes": 245000,
      "views": 12500000,
      "bookmarks": 8200,
      "isRepost": false,
      "hasImage": true,
      "hasVideo": false
    }
  ],
  "scrapedAt": "2026-01-26T10:30:00.000Z",
  "dataSource": "devwithbobby/x-profile-scraper",
  "isPartialData": false
}
```

#### Recent Posts Structure

Each post in the `recentPosts` array contains:

| Field | Type | Description |
|-------|------|-------------|
| `id` | String | Unique tweet ID |
| `text` | String | Tweet content (max 500 chars) |
| `timestamp` | String | ISO timestamp of post |
| `timeAgo` | String | Relative time (e.g., "2h", "3d") |
| `url` | String | Direct link to the tweet |
| `replies` | Integer | Number of replies |
| `reposts` | Integer | Number of reposts/retweets |
| `likes` | Integer | Number of likes |
| `views` | Integer | Number of views |
| `bookmarks` | Integer | Number of bookmarks |
| `isRepost` | Boolean | Whether this is a repost |
| `hasImage` | Boolean | Contains an image |
| `hasVideo` | Boolean | Contains a video |

#### Partial Data (Login Wall)

When X blocks full access, the scraper returns available meta data:

```json
{
  "inputUrl": "https://x.com/someuser",
  "scrapedUrl": "https://x.com/someuser",
  "username": "someuser",
  "displayName": "Some User",
  "bio": "Bio extracted from meta tags",
  "isPartialData": true,
  "scrapedAt": "2026-01-26T10:30:00.000Z"
}
```

#### Error Cases

```json
{
  "inputUrl": "https://x.com/deletedaccount",
  "username": "deletedaccount",
  "error": "Account not found",
  "scrapedAt": "2026-01-26T10:30:00.000Z"
}
```

### Output Fields

| Field | Type | Description |
|-------|------|-------------|
| `inputUrl` | String | Original URL requested |
| `scrapedUrl` | String | Final URL after redirects |
| `username` | String | X handle without @ |
| `displayName` | String | Profile display name |
| `bio` | String | Profile bio/description |
| `location` | String | Location from profile |
| `website` | String | Website URL from profile |
| `joinedDate` | String | When the account was created |
| `followers` | Integer | Number of followers |
| `following` | Integer | Number of accounts following |
| `tweetsCount` | Integer | Total number of tweets |
| `likesCount` | Integer | Total number of likes |
| `isVerified` | Boolean | Legacy verified status |
| `isBlueVerified` | Boolean | X Blue/Premium verified |
| `profilePicture` | String | High-res profile image URL |
| `bannerImage` | String | Banner image URL |
| `recentPosts` | Array | Up to 10 recent posts with engagement metrics |
| `isPartialData` | Boolean | True if some data couldn't be extracted |
| `error` | String | Error message if scrape failed |

### Technical Details

#### Anti-Detection Measures

- **Browser Fingerprinting** - Realistic Chrome/desktop fingerprints via `useFingerprints`
- **Stealth Headers** - Proper `Sec-Ch-Ua`, `Sec-Fetch-*` headers matching real Chrome
- **WebDriver Masking** - Overrides `navigator.webdriver` and other automation indicators
- **Human-like Behavior** - Random delays, scrolling, variable viewport sizes
- **Popup Dismissal** - Automatically closes X login modals and prompts

#### Block Detection

The scraper detects and handles:

- Login redirects
- Suspended accounts
- Deleted/non-existent accounts
- Rate limiting
- Login wall modals

#### Limitations

- **Authentication required** - X requires login to view profiles; cookies must be provided
- **Rate limiting** - X may block after many requests from the same IP
- **Private accounts** - Cannot scrape protected/private profiles
- **Cookie expiration** - Session cookies expire; you may need to re-export periodically

#### Best Practices

1. **Use Residential Proxies** for highest success rates (X heavily blocks datacenter IPs)
2. **Keep concurrency at 1** to avoid triggering rate limits
3. **Refresh cookies weekly** - Export fresh cookies if you see authentication failures
4. **Space out runs** to avoid IP-based blocks
5. **Use low request rates** (10/min or less recommended)

### Proxy Options

| Option | Description | Best For |
|--------|-------------|----------|
| `RESIDENTIAL` | Residential proxies (default) | Highest success rate |
| `BUYPROXIES94952` | Datacenter proxies | Cost-effective, lower success |
| `StaticUS3` | Static US IPs | Consistent identity |

### Cost Estimation

- Residential proxy: ~$12.50 per 1000 requests (recommended)
- Datacenter proxy: ~$0.25 per 1000 requests (may have lower success)
- Compute: ~$0.15 per 100 profiles

### Summary Statistics

After each run, a summary is saved to the key-value store:

```json
{
  "totalRequested": 10,
  "totalScraped": 8,
  "totalErrors": 2,
  "averageFollowers": 1250000,
  "verifiedCount": 3,
  "proxyUsed": "RESIDENTIAL",
  "authenticated": true,
  "completedAt": "2026-01-26T10:35:00.000Z"
}
```

### Troubleshooting

#### "No data could be extracted"

- **Check your cookies** - Make sure you exported all cookies, not just specific ones
- **Refresh cookies** - Log out of X, log back in, and export fresh cookies
- **Verify auth_token** - Ensure the `auth_token` cookie is present in your export

#### "Missing required cookie: auth_token"

- You need to be logged into X when exporting cookies
- Make sure you're on x.com (not twitter.com) when exporting

#### Rate limiting / blocked

- Reduce `maxRequestsPerMinute` to 5
- Use `RESIDENTIAL` proxy type
- Wait a few hours before retrying

### Support

For issues or feature requests, contact the author or open an issue on the actor's page.

# Actor input Schema

## `profiles` (type: `string`):

Enter X/Twitter profiles to extract data from. You can use:

- Just usernames: elonmusk
- With @: @elonmusk
- Full URLs: https://x.com/elonmusk or https://twitter.com/elonmusk

One per line or comma-separated.

## `proxyType` (type: `string`):

Select the proxy type to use. Residential proxies are recommended for X/Twitter due to aggressive anti-bot measures.

## `maxConcurrency` (type: `integer`):

Maximum number of pages to scrape in parallel. Keep very low (1-2) as X/Twitter blocks aggressively.

## `maxRequestsPerMinute` (type: `integer`):

Rate limit for requests. Keep low (5-10) to avoid triggering X/Twitter's anti-bot measures.

## `cookies` (type: `string`):

REQUIRED: X/Twitter requires login to view profiles. Paste your session cookies as JSON array.

How to get cookies:

1. Log into x.com in your browser
2. Install 'Cookie-Editor' browser extension
3. Click the extension icon on x.com
4. Click 'Export' → 'Export as JSON'
5. Paste the entire JSON array here

Required cookies: 'auth_token' and 'ct0' (CSRF token)

Example format:
\[{"name": "auth_token", "value": "xxx", "domain": ".x.com"}, {"name": "ct0", "value": "xxx", "domain": ".x.com"}]

## Actor input object example

```json
{
  "profiles": "elonmusk\nOpenAI",
  "proxyType": "RESIDENTIAL",
  "maxConcurrency": 1,
  "maxRequestsPerMinute": 10
}
```

# Actor output Schema

## `overview` (type: `string`):

Quick overview of scraped profiles with key metrics

## `detailed` (type: `string`):

Full data for all scraped profiles

## `allData` (type: `string`):

Complete raw JSON data

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "profiles": `elonmusk
OpenAI`
};

// Run the Actor and wait for it to finish
const run = await client.actor("devwithbobby/x-profile-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "profiles": """elonmusk
OpenAI""" }

# Run the Actor and wait for it to finish
run = client.actor("devwithbobby/x-profile-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "profiles": "elonmusk\\nOpenAI"
}' |
apify call devwithbobby/x-profile-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devwithbobby/x-profile-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/c8aTqXPSxa0Zdan2g/builds/dozHRezWA7Ckv3qLe/openapi.json