# Google Scholar Scraper (`cloud9_ai/google-scholar-scraper`) Actor

Extract academic papers from Google Scholar: title, authors, year, journal, citation count, abstract snippet, PDF links. Search by keyword with year range filters. Stricter rate limiting for reliability. Perfect for literature review, research trend analysis, citation tracking.

- **URL**: https://apify.com/cloud9_ai/google-scholar-scraper.md
- **Developed by:** [cloud9](https://apify.com/cloud9_ai) (community)
- **Categories:** AI, Automation
- **Features:** Limited permissions
- **Stats:** 19 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Google Scholar Scraper

Apify Actor to scrape Google Scholar search results with advanced filtering options.

### Features

- **Search by keyword**: Find academic papers, articles, and books
- **Author filtering**: Filter results by specific authors
- **Year range**: Limit results to specific publication years
- **Sort options**: Sort by relevance or date
- **Citation data**: Extract citation counts and related articles
- **PDF links**: Automatically detect available PDF downloads
- **Rate limiting**: Built-in 5-10 second delays to respect Google Scholar
- **Robust parsing**: Handles various result formats (articles, books, citations)

### Input Parameters

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `searchQuery` | String | ✅ | Search query (e.g., "machine learning") |
| `author` | String | ❌ | Filter by author name |
| `yearFrom` | Number | ❌ | Publication year start (1900-2100) |
| `yearTo` | Number | ❌ | Publication year end (1900-2100) |
| `sortBy` | Select | ❌ | Sort by "relevance" or "date" (default: "relevance") |
| `includePatents` | Boolean | ❌ | Include patents in results (default: true) |
| `includeCitations` | Boolean | ❌ | Include citations in results (default: true) |
| `maxResults` | Number | ❌ | Maximum results to scrape (default: 100, max: 1000) |

### Output Format

Each result contains:

```json
{
  "title": "Paper title",
  "articleUrl": "https://example.com/paper.pdf",
  "pdfUrl": "https://example.com/download.pdf",
  "authors": "John Doe, Jane Smith",
  "year": 2023,
  "journal": "Journal of Machine Learning Research",
  "abstract": "This paper presents...",
  "citationCount": 42,
  "citedByUrl": "https://scholar.google.com/scholar?cites=...",
  "relatedArticlesUrl": "https://scholar.google.com/scholar?q=related:...",
  "allVersionsCount": 3,
  "isBook": false,
  "isCitation": false,
  "isPdf": true
}
```

### Usage Example

#### Input

```json
{
  "searchQuery": "deep learning natural language processing",
  "author": "Yoshua Bengio",
  "yearFrom": 2020,
  "yearTo": 2024,
  "sortBy": "date",
  "maxResults": 50
}
```

#### Run Locally

```bash
## Install dependencies
npm install

## Build TypeScript
npm run build

## Run actor (requires input.json in root or Apify environment)
npm start
```

### Important Notes

#### Rate Limiting

Google Scholar is **very strict** about automated access:

- Actor uses 5-10 second delays between requests
- Realistic User-Agent rotation
- Proper HTTP headers to mimic browser behavior
- Automatic CAPTCHA detection and graceful shutdown

**Recommendation**:

- Keep `maxResults` under 100 for reliability
- Use longer delays for larger scrapes
- Consider using Google Scholar API alternatives for production use

#### CAPTCHA/Blocking

If Google Scholar detects automation:

- Actor logs a warning and stops gracefully
- No partial results are lost (already scraped data is saved)
- You can retry with longer delays or from a different IP

#### Legal Considerations

- Respect Google Scholar's Terms of Service
- Use for research/academic purposes
- Do not overload their servers
- Consider API alternatives for commercial use

### Development

#### Build

```bash
npm run build
```

#### Local Testing

```bash
npm run dev
```

#### Docker Build

```bash
docker build -t google-scholar-scraper .
docker run -e APIFY_INPUT='{"searchQuery":"machine learning"}' google-scholar-scraper
```

### Troubleshooting

#### No Results Found

- Check if query has typos
- Try broader search terms
- Verify year range is valid

#### CAPTCHA Detected

- Reduce `maxResults`
- Run actor less frequently
- Use different IP address
- Consider Google Scholar API

#### Parser Errors

- Google Scholar HTML structure may change
- Open an issue with example query
- Actor will skip unparseable results

### License

Apache-2.0

### Support

For issues or questions, please open a GitHub issue or contact the Apify support team.

# Actor input Schema

## `searchQuery` (type: `string`):

The search query to search for on Google Scholar

## `author` (type: `string`):

Filter by author name (optional)

## `yearFrom` (type: `integer`):

Publication year start (optional)

## `yearTo` (type: `integer`):

Publication year end (optional)

## `sortBy` (type: `string`):

Sort results by relevance or date

## `includePatents` (type: `boolean`):

Include patents in search results

## `includeCitations` (type: `boolean`):

Include citations in search results

## `maxResults` (type: `integer`):

Maximum number of results to scrape

## Actor input object example

```json
{
  "searchQuery": "machine learning",
  "sortBy": "relevance",
  "includePatents": true,
  "includeCitations": true,
  "maxResults": 100
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "machine learning"
};

// Run the Actor and wait for it to finish
const run = await client.actor("cloud9_ai/google-scholar-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQuery": "machine learning" }

# Run the Actor and wait for it to finish
run = client.actor("cloud9_ai/google-scholar-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "machine learning"
}' |
apify call cloud9_ai/google-scholar-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cloud9_ai/google-scholar-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oqJym5huit0okoiVx/builds/iBvvlzTJgGx0SHUZT/openapi.json