# arXiv Papers Scraper - Preprints, Authors & PDFs (`thirdwatch/arxiv-papers-scraper`) Actor

Search arXiv through its public API. Export paper IDs, titles, abstracts, authors, categories, submission and update dates, DOI, journal reference, comments, and PDF links.

- **URL**: https://apify.com/thirdwatch/arxiv-papers-scraper.md
- **Developed by:** [Thirdwatch](https://apify.com/thirdwatch) (community)
- **Categories:** AI
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Papers Scraper - Preprints, Authors & PDFs

> **Thirdwatch portfolio:** 8,800+ users across 140+ public Actors, 2M+ records delivered, >95% run success. [Explore all Thirdwatch Actors](https://apify.com/thirdwatch).

> Search arXiv papers — titles, authors, abstracts, categories, and PDF links from the open preprint index.

### What you get

Search arXiv papers — titles, authors, abstracts, categories, and PDF links from the open preprint index.

### Why use this arXiv Papers scraper

- **Full paper metadata** — title, authors, abstract, subject categories, published date, and arXiv/PDF links.
- **Free open index** — arXiv's public API requires no key and no browser.
- **Research-grade rows** — one clean record per paper for literature pipelines.

### Scrape arXiv Papers data in minutes

```json
{
  "queries": [
    "large language models"
  ]
}
```

```json
{
  "queries": [
    "transformer architecture"
  ]
}
```

### Input parameters

| Parameter | Required | Description |
|---|---:|---|
| `queries` | Yes | Topics, paper titles, authors, abstracts, or arXiv categories to search. Default: `['large language models']`. |
| `searchField` | No | Apply each query to all metadata or a specific arXiv field. One of: all, title, author, abstract, category. Default: `all`. |
| `sortBy` | No | Order results using arXiv's supported sort fields. One of: relevance, submittedDate, lastUpdatedDate. Default: `submittedDate`. |
| `sortOrder` | No | Return newest or highest values first with descending order, or reverse them. One of: descending, ascending. Default: `descending`. |
| `maxResultsPerQuery` | No | Stop after this many unique papers for each search term. Default: `100`. |

### Output fields

- `arxiv_id`
- `title`
- `summary`
- `published`
- `updated`
- `authors`
- `categories`
- `primary_category`
- `doi`
- `journal_reference`
- `comment`
- `paper_url`
- `pdf_url`
- `query`
- `source`

#### Example result

```json
{
  "arxiv_id": "2608.27458v1",
  "title": "The Atacama Cosmology Telescope: Constraints on the anisotropic screening and birefringence effects with DR6",
  "summary": "While we have learned much from recent surveys of the mm-wave sky, many secondary anisotropies of the CMB remain under-e…",
  "published": "2026-08-27T17:59:59Z",
  "updated": "2026-08-27T17:59:59Z",
  "authors": [
    "Darby M. Kramer",
    "Alexander van Engelen",
    "Frank J. Qu",
    "…"
  ],
  "categories": [
    "astro-ph.CO"
  ],
  "primary_category": "astro-ph.CO",
  "doi": null,
  "journal_reference": null,
  "comment": "31 pages, 24 figures. To be submitted to ApJ",
  "paper_url": "http://arxiv.org/abs/2608.27458v1",
  "pdf_url": "https://arxiv.org/pdf/2608.27458v1",
  "query": "large language models",
  "source": "arXiv"
}
```

### Common use cases

- Monitor new papers in a field
- Build literature datasets
- Feed summarization and RAG pipelines
- Track authors and labs publishing on arXiv

### Compared to alternatives

- **vs other arXiv Papers scrapers on the Store** — pay-per-result billing means empty or failed searches cost nothing; you are not charged for a run that returns zero rows.
- **vs building your own** — no proxy management, markup-change maintenance, or infrastructure to run; paste inputs and consume the dataset.

### Limitations

- arXiv only — paywalled venues need institutional access.
- Abstracts are included; full PDF text is linked, not extracted.
- The public API rate-limits heavy pagination — bound runs with `maxResults`.

### FAQ

**Do I need an account or cookies?**

No — the Actor reads public data only. There is no login to configure and no account to risk.

**What happens when a search returns nothing?**

The run completes with zero (or an explicit error) rows. Per-result billing means you are not charged for empty output.

**Can I run it on a schedule?**

Yes — attach an Apify schedule or use the saved tasks to re-run the same inputs daily or weekly.

### Related guides and Actors

- [Analyze Arxiv Papers With Python](https://thirdwatch.dev/blog/analyze-arxiv-papers-with-python)
- [Build Arxiv Research Dataset](https://thirdwatch.dev/blog/build-arxiv-research-dataset)
- [Monitor New Ai Papers On Arxiv](https://thirdwatch.dev/blog/monitor-new-ai-papers-on-arxiv)

Last verified: 2026-09

More scrapers at [thirdwatch.dev](https://thirdwatch.dev).

# Actor input Schema

## `queries` (type: `array`):

Topics, paper titles, authors, abstracts, or arXiv categories to search.

## `searchField` (type: `string`):

Apply each query to all metadata or a specific arXiv field.

## `sortBy` (type: `string`):

Order results using arXiv's supported sort fields.

## `sortOrder` (type: `string`):

Return newest or highest values first with descending order, or reverse them.

## `maxResultsPerQuery` (type: `integer`):

Stop after this many unique papers for each search term.

## Actor input object example

```json
{
  "queries": [
    "large language models"
  ],
  "searchField": "all",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResultsPerQuery": 10
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "large language models"
    ],
    "searchField": "all",
    "sortBy": "submittedDate",
    "sortOrder": "descending",
    "maxResultsPerQuery": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("thirdwatch/arxiv-papers-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["large language models"],
    "searchField": "all",
    "sortBy": "submittedDate",
    "sortOrder": "descending",
    "maxResultsPerQuery": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("thirdwatch/arxiv-papers-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "large language models"
  ],
  "searchField": "all",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResultsPerQuery": 10
}' |
apify call thirdwatch/arxiv-papers-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thirdwatch/arxiv-papers-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tJanxpNvubDG8Eo7M/builds/vEGasIsPZfA7Jon6f/openapi.json