# arXiv Search Scraper 📚 (`easyapi/arxiv-search-scraper`) Actor

Extract comprehensive research paper data from arXiv search results. Get detailed metadata including titles, authors, abstracts, categories and more. Perfect for academic research monitoring, trend analysis and building paper databases. 🎓📚

- **URL**: https://apify.com/easyapi/arxiv-search-scraper.md
- **Developed by:** [EasyApi](https://apify.com/easyapi) (community)
- **Categories:** Integrations, Other
- **Features:** Limited permissions
- **Stats:** 54 total users, 3 monthly users, 100.0% runs succeeded, 4 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Search Scraper 📚

Scrape research papers, authors, and metadata from arXiv search results. Get detailed information about academic papers including titles, authors, abstracts, categories, submission dates and more.

### Features ✨

- 🔍 Scrape papers from any arXiv search URL
- 📄 Extract comprehensive paper metadata including:
  - Paper ID and PDF links
  - Title and abstract
  - Author names and profile URLs
  - Research categories and classifications
  - Submission dates and comments
- ⚡ Fast and efficient pagination handling
- 🔄 Support for multiple search URLs
- ⚙️ Configurable maximum items limit
- 🌐 Proxy support for reliable scraping

### Use Cases 💡

- Research trend analysis
- Academic paper monitoring
- Building paper databases
- Author tracking
- Category-based paper collection
- Literature review automation

### Input Parameters 🎛️

The actor accepts the following input parameters:

| Field | Type | Description |
|-------|------|-------------|
| searchUrls | Array | List of arXiv search URLs to scrape |
| maxItems | Integer | Maximum number of items to scrape (optional) |
| proxyConfiguration | Object | Proxy settings (optional) |

### Output 📊

The actor stores results in dataset with the following fields for each paper:

- `searchUrl`: Source search URL
- `arxivId`: Unique arXiv paper ID
- `pdfUrl`: Direct link to PDF
- `categories`: Research categories with codes and names
- `title`: Paper title
- `authors`: Author details including names and profile URLs
- `abstract`: Full paper abstract
- `submissionDate`: Paper submission date
- `comments`: Additional paper comments

### Example Usage 💻

#### Input Example

A full explanation of an input example in JSON.

```json
{
    "searchUrls": [
        "https://arxiv.org/search/?query=ai&searchtype=all&source=header"
    ],
    "maxItems": 60
}
```

#### Output sample

The results will be wrapped into a dataset which you can always find in the **Storage** tab. Here's an excerpt from the data you'd get if you apply the input parameters above:

And here is the same data but in JSON. You can choose in which format to download your data: JSON, JSONL, Excel spreadsheet, HTML table, CSV, or XML.

```json
[
    {
        "searchUrl": "https://arxiv.org/search/?query=ai&searchtype=all&source=header",
        "arxivId": "arXiv:2502.21286",
        "pdfUrl": "https://arxiv.org/pdf/2502.21286",
        "categories": [
            {
                "code": "cs.CR",
                "name": "Cryptography and Security"
            },
            {
                "code": "cs.LG",
                "name": "Machine Learning"
            },
            {
                "code": "cs.NI",
                "name": "Networking and Internet Architecture"
            },
            {
                "code": "doi"
            },
            {
                "code": "10.1109/TNSM.2024.3376631"
            }
        ],
        "title": "Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis",
        "authors": [
            {
                "name": "Li Yang",
                "url": "https://arxiv.org/search/?searchtype=author&query=Yang%2C+L"
            },
            {
                "name": "Mirna El Rajab",
                "url": "https://arxiv.org/search/?searchtype=author&query=Rajab%2C+M+E"
            },
            {
                "name": "Abdallah Shami",
                "url": "https://arxiv.org/search/?searchtype=author&query=Shami%2C+A"
            },
            {
                "name": "Sami Muhaidat",
                "url": "https://arxiv.org/search/?searchtype=author&query=Muhaidat%2C+S"
            }
        ],
        "abstract": "Zero-Touch Networks (ZTNs) represent a state-of-the-art paradigm shift towards fully automated and intelligent network management, enabling the automation and intelligence required to manage the complexity, scale, and dynamic nature of next-generation (6G) networks. ZTNs leverage Artificial Intelligence (AI) and Machine Learning (ML) to enhance operational efficiency, support intelligent decision-making, and ensure effective resource allocation. However, the implementation of ZTNs is subject to security challenges that need to be resolved to achieve their full potential. In particular, two critical challenges arise: the need for human expertise in developing AI/ML-based security mechanisms, and the threat of adversarial attacks targeting AI/ML models. In this survey paper, we provide a comprehensive review of current security issues in ZTNs, emphasizing the need for advanced AI/ML-based security mechanisms that require minimal human intervention and protect AI/ML models themselves. Furthermore, we explore the potential of Automated ML (AutoML) technologies in developing robust security solutions for ZTNs. Through case studies, we illustrate practical approaches to securing ZTNs against both conventional and AI/ML-specific threats, including the development of autonomous intrusion detection systems and strategies to combat Adversarial ML (AML) attacks. The paper concludes with a discussion of the future research directions for the development of ZTN security approaches.\n        △ Less",
        "submissionDate": "28 February, 2025",
        "comments": "Published in IEEE Transactions on Network and Service Management (TNSM); Code is available at Github link: https://github.com/Western-OC2-Lab/AutoML-and-Adversarial-Attack-Defense-for-Zero-Touch-Network-Security"
    },
    ...
]
```

### Related Actors

- 🔬 [Nature Search Results Scraper](https://apify.com/easyapi/nature-search-results-scraper) - Extract comprehensive research article data from Nature.com search results
- 📚 [Goodreads Book Scraper](https://apify.com/easyapi/goodreads-book-scraper) - Extract comprehensive book data for literature research and analysis
- 📚 [Goodreads Review Scraper](https://apify.com/easyapi/goodreads-review-scraper) - Extract detailed book reviews and ratings for academic literature analysis
- 📚 [Udemy Course Scraper](https://apify.com/easyapi/udemy-course-scraper) - Extract detailed course information for educational content research
- 📚 [Udemy Course Reviews Scraper](https://apify.com/easyapi/udemy-course-reviews-scraper) - Collect comprehensive course review data for educational analysis
- 📄 [Article Content Extractor](https://apify.com/easyapi/article-content-extractor) - Extract clean article content and metadata from any web page
- 🔍 [Google Scholar Scraper](https://apify.com/easyapi/google-scholar-scraper) - Collect scholarly results with flexible search options and citation filtering
- 🔍 [AI-powered Search](https://apify.com/easyapi/ai-powered-search) - Get AI-enhanced search summaries with references and optimization tips
- 📊 [Text Sentiment Analysis](https://apify.com/easyapi/text-sentiment-analysis) - Analyze sentiment in research abstracts and academic content
- 📝 [Text Summarization](https://apify.com/easyapi/text-summarization) - Generate concise summaries of research papers and documents
- 🔍 [PubMed Search Scraper](https://apify.com/easyapi/pubmed-search-scraper) - Extract research papers and academic articles from PubMed
- 📚 [Substack Publications Scraper](https://apify.com/easyapi/substack-publications-scraper) - Collect detailed academic newsletter and publication data
- 📚 [Substack Posts Scraper](https://apify.com/easyapi/substack-posts-scraper) - Extract comprehensive academic post and article content
- 🔍 [Keyword Discovery Tool](https://apify.com/easyapi/keyword-discovery-tool) - Discover relevant academic keywords and research topics
- 🔍 [Keyword Density Checker](https://apify.com/easyapi/keyword-density-checker) - Analyze keyword frequency in academic content
- 📚 [Medium Posts Search Scraper](https://apify.com/easyapi/medium-posts-search-scraper) - Extract detailed article data for research content analysis

# Actor input Schema

## `searchUrls` (type: `array`):

List of arXiv search URLs to scrape

## `maxItems` (type: `integer`):

Maximum number of items to scrape

## `proxyConfiguration` (type: `object`):

Proxy settings

## Actor input object example

```json
{
  "searchUrls": [
    "https://arxiv.org/search/?query=ai&searchtype=all&source=header"
  ],
  "maxItems": 60,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("easyapi/arxiv-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("easyapi/arxiv-search-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call easyapi/arxiv-search-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,easyapi/arxiv-search-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3ipocAoaHICS3cVdB/builds/s1rH9G1cEXfVfb9c5/openapi.json