Academic Paper Resolver — DOI/arXiv to Citations for AI

Academic Paper Scraper — DOI, arXiv & PubMed to Citations, Abstracts & Metadata

Turn any DOI, arXiv ID, PubMed ID, or paper title into a rich, structured record — authors, abstract, venue, year, citation count, references, open-access PDF, topics, keywords and cross-IDs — merged from several scholarly databases into one clean answer, plus a RAG-ready Markdown card. This high-throughput academic paper scraper resolves messy or mixed identifiers into verified bibliographic metadata, so you never have to copy-paste from a dozen journal pages again. No login, no API key, no coding.

Export every resolved paper to JSON, CSV, Excel, or API.

Built for AI research agents, RAG pipelines, literature-review tools, reference managers, bibliometrics teams, and data journalists who need trustworthy, machine-readable scholarly metadata on demand. Resolution is completely unlocked for 1,000+ papers, scaling seamlessly across large bibliographies.

---

⚡ At a glance

Metric / Feature	Specification
Data Sources	Merged across OpenAlex, Crossref, arXiv, and Semantic Scholar
Accepted Identifiers	DOIs (10.xxx), arXiv IDs (1706.03762), PubMed IDs (PMID), and plain English paper titles
Output Artifacts	Flat structured metadata JSON, titled reference arrays, and RAG-ready Markdown cards
Impact Metrics	Total citations, Field-Weighted Citation Impact (FWCI), and top-1% / top-10% percentile flags
Resolution Scale	1,000+ papers unlocked — resolves large-scale corpora with parallel 5-worker concurrency
Throughput / Speed	~5 papers resolved per second; 100 papers in ~20 seconds
Recommended Memory	512 MB – 1024 MB (pure HTTP, zero headless browser overhead)
Output Formats	JSON, CSV, Excel (XLSX), XML, HTML
Integrity Checks	Automatic retraction detection with prominent warning banners

---

🔎 Why scrape academic papers?

Scholarly metadata is heavily fragmented. A single research paper's record is scattered across multiple repositories — OpenAlex, Crossref, arXiv, PubMed, and institutional archives — each using different schemas, persistent identifiers, and coverage standards. One repository provides the abstract but lacks reference titles; another provides titled references but omits citation impact; a third hosts preprint PDFs but lacks publication metadata. Reconciling these manually across hundreds of papers is tedious and error-prone.

Furthermore, generative AI models (LLMs) cannot reliably hallucinate accurate citation statistics, exact publication years, co-author lists, or formal retraction statuses. Autonomous scientific workflows require rigorous grounding in verified bibliographic data. This academic paper scraper provides that foundation: input an identifier or title, and receive an enriched, multi-source record suitable for downstream retrieval-augmented generation (RAG) and citation graphing.

Because this actor automatically federates data from four primary scholarly hubs, gaps in one source are dynamically resolved by another:

• OpenAlex: Core bibliographic graph, normalized topics, concept tags, and field-weighted impact.
• Crossref: Titled, dated reference lists and publisher-deposited abstracts.
• arXiv: Preprint abstract text and direct open-access PDF URLs.
• Semantic Scholar: Natural language TL;DR summaries and influential citation counts.

Manual Literature Search vs. Automated Paper Resolution

Evaluation Factor	Manual Journal Browsing	Academic Paper Scraper
Speed	3–5 minutes per paper	~5 papers per second (automated)
Batch Capacity	10–20 papers per session	1,000+ papers per execution
Cross-ID Resolution	Manual searching across 4 databases	Auto-detects DOI, arXiv, PMID, and titles
Abstract Decoding	Manual copying behind paywalls	Reconstructed text & Crossref backfills
Citation Metrics	Outdated or platform-locked counts	Live multi-source citation & FWCI counts
RAG Artifacts	Manual note-taking	Auto-generated Markdown cards with links

High-Value Reasons to Extract Academic Paper Data

• Retrieval-Augmented Generation (RAG): Populate vector databases with normalized abstracts and reference lists to prevent LLM hallucinations.
• Autonomous Research Agents: Equip agents (via MCP or REST API) with a keyless verification tool to validate research claims in real time.
• Systematic Literature Reviews: Ingest extensive lists of DOIs from screening tools (Covidence, Rayyan) and export enriched spreadsheets with one click.
• Citation Graph Traversal: Explore two-way citation trees by extracting both cited references and forward citing works.
• Research Integrity & Retraction Auditing: Automatically screen bibliographies against retraction notices before submitting manuscripts or releasing models.
• Open-Access Discovery: Immediately identify open-access papers and direct links to publisher-hosted and repository PDFs.

---

✨ Key Features & Capabilities

• ✅ Universal Identifier Auto-Detection: Automatically classifies DOIs, arXiv identifiers, PubMed PMIDs, and title strings without requiring manual tagging.
• ✅ Multi-Database Federation: Joins OpenAlex, Crossref, arXiv, and Semantic Scholar into a single consolidated, authoritative output object.
• ✅ 1,000+ Batch Resolution Unlocked: Handles massive bibliographic collections without arbitrary result slicing or hard-coded limits.
• ✅ Inverted-Index Abstract Reconstruction: Translates OpenAlex's compressed inverted index representations into clean, readable paragraph text.
• ✅ Full Titled Reference Lists: Extracts structured reference arrays containing titles, publication years, authors, and DOIs.
• ✅ Forward Citing Papers: Enabling includeCitations fetches forward citation links to map evolving scientific discourse.
• ✅ Advanced Bibliometrics (FWCI): Provides Field-Weighted Citation Impact where 1.0 represents the global field baseline, alongside top-1% and top-10% indicators.
• ✅ Retraction Alert Banners: Automatically inspects retraction statuses and embeds prominent warnings into both the structured JSON and Markdown outputs.
• ✅ RAG-Ready Markdown Cards: Outputs a pre-formatted Markdown summary for every paper, ready for direct prompt injection or document embedding.
• ✅ Keyless & Polite Execution: Operates Keyless with exponential back-off and polite request throttling across all upstream scholarly endpoints.

---

🎯 Primary Use Cases

1. AI Research & Retrieval Engineers

Ground LLM agents in verified peer-reviewed publications. Inject the generated markdown artifact into conversational prompts to cite exact authors, venues, and DOIs.

2. Systematic Review & Meta-Analysis Teams

Process hundreds of screening hits simultaneously. Normalize publication dates, author lists, and open-access PDF links for rapid evidence synthesis.

3. Bibliometrics & Science Policy Analysts

Benchmark research group performance using normalized FWCI metrics rather than raw citation totals that favor older publications.

4. University Libraries & Institutional Repositories

Audit institutional publications, verify open-access compliance status (gold, green, hybrid), and aggregate direct repository links.

5. Scientific Journalists & Science Communicators

Fact-check breaking science stories, verify primary literature citations, and confirm whether cited papers remain in good standing.

---

📥 Input Parameters

Configure the actor easily using the visual form in Apify Console or pass a JSON payload via the REST API.

Field	Type	Default	Description
queries	Array	["10.1038/nature14539", "1706.03762", "attention is all you need"]	List of DOIs, arXiv IDs, PubMed PMIDs, or paper titles to resolve. Leave empty to run a built-in landmark sample.
includeReferences	Boolean	true	When true, includes the paper's deposited reference list in the output.
includeCitations	Boolean	false	When true, fetches forward citing papers (requires an extra API lookup per paper).
maxReferences	Integer	50	Maximum number of references and citing papers to retain per record. Set to 0 for unlimited.
maxResults	Integer	0	Safety limit on total papers resolved in a single run. Set to 0 for unlimited (1,000+ items unlocked).
proxyConfiguration	Object	{"useApifyProxy": false}	Proxy settings. Not required as scholarly databases are open public APIs.

Input Examples

Example 1: Multi-Identifier Batch Resolution (Prefill)

{
  "queries": [
    "10.1038/nature14539",
    "1706.03762",
    "attention is all you need"
  ],
  "includeReferences": true,
  "includeCitations": false,
  "maxReferences": 50,
  "maxResults": 100
}

Example 2: Deep 1,000+ DOI Corpus Resolution

{
  "queries": [
    "10.1103/PhysRevLett.116.061102",
    "10.1126/science.1258096"
  ],
  "includeReferences": true,
  "includeCitations": true,
  "maxReferences": 100,
  "maxResults": 1000
}

---

📤 Output Dataset & Field Coverage

The actor outputs clean, flat JSON objects where each record corresponds to a resolved scholarly publication.

Field Specification & Coverage Indicators

📄 Core Bibliographic Identity

Field	Type	Coverage	Description
query	String	🟢 High (100%)	Original identifier or title submitted for resolution.
resolved	String	🟢 High (100%)	Status string ("true" if found, "false" if unresolved).
title	String	🟢 High (100%)	Official canonical publication title.
authors	String	🟢 High (~99%)	Comma-delimited list of author names.
year	String	🟢 High (100%)	Publication year.
publicationDate	String	🟢 High (~96%)	ISO publication date (YYYY-MM-DD).
venue	String	🟢 High (~93%)	Journal, conference proceeding, or repository host.
publisher	String	🟢 High (~91%)	Academic publishing house (e.g. Elsevier, Springer Nature).
type	String	🟢 High (100%)	Publication type (article, preprint, book-chapter).
doi	String	🟢 High (~95%)	Digital Object Identifier without URL prefixes.
arxivId	String	🟡 Conditional (~25%)	arXiv preprint accession number.
pmid	String	🟡 Conditional (~42%)	PubMed identifier for biomedical publications.

📊 Impact Metrics & Citation Stats

Field	Type	Coverage	Description
citationCount	String	🟢 High (100%)	Total global citation count across literature databases.
referenceCount	String	🟢 High (100%)	Total number of references deposited by authors.
influentialCitationCount	String	🟡 Conditional (~20%)	High-impact citations calculated by Semantic Scholar.
fwci	String	🟢 High (~90%)	Field-Weighted Citation Impact (1.0 = world baseline).
citationPercentileTop1	String	🟢 High (~90%)	"true" if paper ranks in the top 1% most cited for year & field.
citationPercentileTop10	String	🟢 High (~90%)	"true" if paper ranks in the top 10% most cited for year & field.

🔓 Open Access & PDF Delivery

Field	Type	Coverage	Description
isOpenAccess	String	🟢 High (100%)	Open access boolean flag ("true" / "false").
oaStatus	String	🟢 High (~98%)	Classification: "gold", "green", "hybrid", "bronze", "closed".
pdfUrl	String	🟡 Conditional (~57%)	Direct link to legal open-access full-text PDF document.
isRetracted	String	🟢 High (100%)	Retraction status flag ("true" / "false").

🤖 RAG Artifacts & Reconstructed Text

Field	Type	Coverage	Description
abstract	String	🟢 High (~86%)	Full plain-text abstract reconstructed from inverted indexes.
concepts	String	🟢 High (100%)	Comma-delimited list of normalized subject taxonomy terms.
primaryTopic	String	🟢 High (~92%)	Granular primary research domain classification.
keywords	String	🟢 High (~88%)	Author and algorithmic keyword phrases.
tldr	String	🟡 Conditional (~25%)	One-sentence summary distilled by Semantic Scholar.
markdown	String	🟢 High (100%)	Self-contained RAG-ready document card with links and headers.
references	Array	🟡 Conditional (~57%)	Structured objects of works cited (title, year, doi, author).
citations	Array	🟡 Conditional	Structured objects of forward citing works when enabled.
sources	String	🟢 High (100%)	Contributing upstream databases (e.g. "OpenAlex, Crossref").
scrapedAt	String	🟢 High (100%)	ISO 8601 resolution timestamp.

Sample Output Record

{
  "query": "10.1038/nature14539",
  "resolved": "true",
  "title": "Deep learning",
  "authors": "Yann LeCun, Yoshua Bengio, Geoffrey Hinton",
  "year": "2015",
  "publicationDate": "2015-05-27",
  "venue": "Nature",
  "publisher": "Springer Nature",
  "type": "article",
  "doi": "10.1038/nature14539",
  "arxivId": "",
  "pmid": "26017442",
  "openAlexId": "https://openalex.org/W2139823423",
  "citationCount": "68420",
  "referenceCount": "102",
  "influentialCitationCount": "4120",
  "fwci": "52.418",
  "citationPercentileTop1": "true",
  "citationPercentileTop10": "true",
  "isOpenAccess": "true",
  "oaStatus": "bronze",
  "pdfUrl": "https://www.nature.com/articles/nature14539.pdf",
  "isRetracted": "false",
  "abstract": "Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction...",
  "concepts": "Artificial intelligence, Machine learning, Computer science, Artificial neural network, Deep learning",
  "primaryTopic": "Deep Learning Techniques and Applications",
  "keywords": "deep learning, neural networks, representation learning, backpropagation",
  "language": "en",
  "tldr": "This review describes the key ideas of deep learning, explains its historical background, and discusses future developments.",
  "url": "https://doi.org/10.1038/nature14539",
  "markdown": "# Deep learning\n\n**Yann LeCun, Yoshua Bengio, Geoffrey Hinton · 2015 · Nature**\n\nCited by 68420 · 102 references · Open access\n\n[DOI](https://doi.org/10.1038/nature14539) · [PDF](https://www.nature.com/articles/nature14539.pdf)\n\n> **TL;DR** This review describes the key ideas of deep learning...\n\n## Abstract\n\nDeep learning allows computational models that are composed of multiple processing layers...",
  "references": [
    {
      "title": "Learning representations by back-propagating errors",
      "year": "1986",
      "doi": "10.1038/323533a0",
      "author": "Rumelhart"
    }
  ],
  "citations": [],
  "sources": "OpenAlex, Crossref, Semantic Scholar",
  "scrapedAt": "2026-11-10T16:28:00.000Z"
}

---

🔁 Scale, Architecture & Resilience

• Multi-Worker Concurrency: Runs parallel resolution workers (concurrency factor of 5) that balance request velocity while observing polite upstream pool guidelines (mailto= courtesy tagging).
• Exponential Back-Off: Automatically pauses upon encountering HTTP 429 rate limits, scaling back-off intervals dynamically.
• Fail-Safe Operation: In the event an invalid DOI or non-existent paper title is passed, the actor marks the record as resolved: "false" rather than throwing unhandled exceptions, ensuring the remaining batch continues smoothly.
• 1,000+ Paper Corpora: Uncapped execution allows users to supply extensive lists of citations, resolving hundreds to thousands of records in minutes.

---

💻 Programmatic Integration & AI Agent Tool Calling

Node.js (Apify SDK)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('haketa/academic-resolver').call({
    queries: ['10.1038/nature14539', '1706.03762'],
    includeReferences: true,
    maxReferences: 25,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const paper of items) {
    console.log(`${paper.title} (${paper.year}) — FWCI: ${paper.fwci}`);
}

Python (Apify Client)

import os
from apify_client import ApifyClient

client = ApifyClient(os.getenv('APIFY_TOKEN'))

run = client.actor('haketa/academic-resolver').call(run_input={
    'queries': ['attention is all you need', '10.1126/science.1258096'],
    'includeReferences': True,
    'maxResults': 50,
})

for paper in client.dataset(run['defaultDatasetId']).list_items().items:
    print(f"[{paper['doi']}] {paper['title']} — Cited {paper['citationCount']} times")

Direct cURL Request

curl -X POST "https://api.apify.com/v2/acts/haketa~academic-resolver/runs" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "queries": ["1706.03762", "10.1038/nature14539"],
    "includeReferences": true
  }'

---

💡 Practical Pro-Tips

• Prioritize Persistent Identifiers: Always pass DOIs (10.xxx) or arXiv IDs whenever possible. Text title searches rely on fuzzy algorithmic matching and can occasionally return adjacent workshop papers with similar names.
• RAG Ingestion: Use the pre-built markdown field directly as your vector database document chunk. It contains titles, authors, citations, topics, and abstracts pre-formatted with clean markdown headers.
• Retraction Screening: Filter exported datasets where isRetracted === "true" to immediately identify compromised publications across your literature base.
• Expanding Citation Networks: Run initial landmark papers with includeCitations: true to discover the most recent breakthrough works citing those seminal discoveries.

---

📚 Reference Managers & Export Integrations

Datasets exported from the Academic Paper Scraper integrate directly with major reference managers and research tools:

Tool / Platform	Recommended Format	Use Case
Zotero	CSV / JSON	Bulk import of resolved bibliographic records and direct open-access PDF links
Mendeley	CSV	Rapid library population with normalized author, year, and venue tags
EndNote	XML / CSV	Large-scale institutional bibliography deduplication and citation formatting
Notion & Obsidian	Markdown	Direct embedding of auto-generated paper cards into personal research knowledge bases
Pinecone / Weaviate	JSON	Vector embedding ingestion using reconstructed abstracts and concept keywords

---

❓ Frequently Asked Questions (FAQ)

Do I need institutional journal subscriptions to use this actor?

No. The actor collects open bibliographic metadata, abstracts, and open-access links from public repositories. You do not need any publisher subscriptions or university proxy logins.

Can I submit a mix of DOIs, arXiv IDs, and paper titles in one run?

Yes. The actor inspects each query string dynamically, identifies whether it represents a DOI, arXiv ID, PubMed PMID, or plain text title, and routes it to the appropriate resolution pipeline.

Why are some abstracts missing?

Certain commercial publishers (e.g., specific legacy Nature or Elsevier journals) do not deposit open abstracts into Crossref or OpenAlex. When abstracts are proprietary, the actor backfills from arXiv or Semantic Scholar where available.

What does Field-Weighted Citation Impact (FWCI) mean?

FWCI compares a paper's actual citation count to the global average citations received by similar papers in the same field and published in the same year. An FWCI of 1.0 represents world average; an FWCI of 2.5 means the paper has received 2.5 times more citations than expected.

---

📋 Changelog

v1.2 (2026-11)

• Unlocked 1,000+ Batch Resolution: Enabled high-throughput resolution of 1,000+ paper queries in a single execution without arbitrary slicing or pagination caps.
• Enhanced Inverted-Index Reconstruction: Upgraded abstract decoding algorithm to handle irregular word-position arrays.
• Improved Crossref & arXiv Fallbacks: Robust fallback chain ensuring maximum abstract and PDF fill rates.
• Fast Prefill Execution: Landmark prefill sample resolves in under 15 seconds on Apify platform.

v1.1 (2026-07)

• Field-Weighted Citation Impact (FWCI): Added normalized bibliometrics and top 1% / top 10% percentile indicators.
• Retraction Banner Support: Embedded retraction alerts directly into RAG-ready Markdown cards.

v1.0 (2026-07)

• Initial release with multi-identifier auto-detection, Crossref references join, and multi-format exports.