# PubMed Scraper for RAG: Papers as Chunked JSON (`getascraper/pubmed-rag-extractor`) Actor Scrape PubMed citations by search term, MeSH, and article type. Returns RAG-ready JSON with full-text chunks from PMC Open Access (cl100k_base, 512/50) and abstract fallback. Drop-in for LangChain, LlamaIndex, Qdrant, Pinecone, Weaviate, pgvector. Skip GROBID / Pubmed Parser. - **URL**: https://apify.com/getascraper/pubmed-rag-extractor.md - **Developed by:** [GetAScraper](https://apify.com/getascraper) (community) - **Categories:** AI, Developer tools, Agents - **Features:** Limited permissions - **Stats:** 7 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.53 / 1,000 literature records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## 🧬 PubMed Scraper for RAG: Papers as Chunked JSON
Turn PubMed biomedical literature into RAG-ready chunks in one call. Search by free text, MeSH terms, or article type, fetch PMC Open Access full text where it exists, and get 512-token chunks with full metadata.
RESEARCH AND REGULATORY RAG EXTRACTORS   •  Turns arXiv, bioRxiv/medRxiv, PubMed, CourtListener case law, and SEC EDGAR filings into RAG-ready, chunked JSON for grounded LLM retrieval.
 PubMed RAG
➤ You are here 		 SEC EDGAR
10-K/10-Q/8-K, chunked 		 arXiv RAG
arXiv papers, chunked 		 bioRxiv/medRxiv
Preprints, chunked JSON
PubMed literature, delivered as clean chunked JSON
Fetches PMC Open Access full text where it exists, falls back to the abstract, and splits everything into tokenizer-aware chunks ready to embed.
🔬 MeSH + article-type filters
Search by MeSH ontology or narrow to reviews 		📖 PMC full text when it exists
NXML stripped to readable prose, abstract fallback 		🧩 Tokenizer-aware chunks
512 tokens, 50 overlap, cl100k_base 		🗄️ Vector-DB neutral
Drops into Qdrant, Pinecone, Weaviate, Chroma
**Scrape PubMed biomedical research into RAG-ready JSON in one call.** Pulls papers by free-text search, MeSH ontology, article type, and date range. Fetches PMC Open Access full-text when available and falls back to the abstract otherwise. Returns fixed-token chunks (cl100k_base, 512 tokens / 50 overlap) with full metadata, ready to drop into **LangChain, LlamaIndex, Qdrant, Pinecone, Weaviate, pgvector, or Chroma**. Built for AI training data teams, pharma/biotech researchers, and anyone who's written "just one more PubMed Parser patch" at 2 AM. ### 🔍 What does PubMed Scraper for RAG do? This Apify Actor scrapes **[PubMed](https://pubmed.ncbi.nlm.nih.gov/)** citations matching your search criteria, fetches PMC Open Access full-text where available, and splits the resulting plain text into **tokenizer-aware chunks (512 tokens, 50-token overlap, tiktoken cl100k_base)** ready to embed or feed into a RAG index. Each output record contains clean metadata (PMID, PMCID, DOI, title, authors, journal, MeSH terms, article types, publication date) and a `chunks` array of `{ idx, text, tokens }` ready for direct ingestion into a vector database. **Try it in the Apify Console.** Fill in a search term (e.g. `SARS-CoV-2 vaccines`), optional MeSH terms or article types, a date range, a paper cap, and hit Start. Download results as JSON, CSV, or Excel. Built on the Apify platform, you also get: scheduled runs, HTTP API access, integrations with Zapier / n8n / Make, proxy rotation, monitoring, and alerts. No infrastructure to run yourself. ### 💡 Why use PubMed Scraper for RAG? - **Skip the PubMed XML parsing grind.** Clean Paper records, not raw E-utilities `` trees. - **PMC Open Access full-text when it exists.** Around 20-30% of PubMed articles are in PMC OA; this Actor grabs NXML and strips it to readable prose. Abstract fallback otherwise. - **MeSH + article-type filtering built in.** Search by MeSH ontology (e.g. `Neoplasms`) or limit to `Review` / `Meta-Analysis` / `Clinical Trial`. - **Pre-chunked for RAG.** `tiktoken cl100k_base` tokenization, compatible with OpenAI `text-embedding-3`, Claude, Cohere, and most BGE/E5/nomic embedding models. - **Vector-DB neutral.** Drop into **Qdrant, Pinecone, Weaviate, pgvector (Supabase / Neon), Chroma, Milvus** without reformatting. - **Framework-ready.** Works with **LangChain, LlamaIndex, Haystack, LangGraph**. - **Respects NCBI rate limits.** 3 requests per second without an API key, 10 per second with one. - **Cheap.** $0.02 per paper. A week of clinical-trial reviews (~200 papers) costs around $4. ### 🚀 How to use PubMed Scraper for RAG
STEP 1
Set your search
Free-text searchTerm, optional meshTerms/articleTypes, and a date range. 		STEP 2
Run the extractor
Fetches PMC Open Access full text where available, falls back to abstract. 		STEP 3
Download chunked JSON
Get token-aware chunks ready to embed, from the Storage tab.
### 📥 Input | Field | Type | Required | Default | Description | |---|---|---|---|---| | `searchTerm` | string | | `SARS-CoV-2 vaccines` | Free-text PubMed query, supports `[mh]`, `[ti]`, `[pt]` syntax | | `meshTerms` | string\[] | | `[]` | MeSH headings. Each entry wrapped with `[mh]`, OR-combined | | `articleTypes` | string\[] | | `[]` | Enum values: clinical_trial / review / meta_analysis / case_report / observational_study | | `dateFrom` | string (YYYY-MM-DD) | | `2024-01-01` | Inclusive publication-date lower bound | | `dateTo` | string (YYYY-MM-DD) | | `2024-01-15` | Inclusive publication-date upper bound | | `maxPapers` | integer | | `50` | Hard cap on papers returned (1 to 100000) | | `ncbiApiKey` | string (secret) | | | Optional NCBI API key for 10 req/s instead of 3 | **Example input:** ```json { "searchTerm": "SARS-CoV-2 vaccines", "articleTypes": ["review", "meta_analysis"], "dateFrom": "2024-01-01", "dateTo": "2024-01-31", "maxPapers": 100 } ``` ### 📤 Output Each paper becomes one dataset item. You can download the dataset in JSON, HTML, CSV, or Excel. ```json { "pmid": "38012345", "pmcid": "PMC10123456", "doi": "10.1038/s41586-024-01234-5", "title": "Durable protection against SARS-CoV-2 variants after boosting", "abstract": "...", "authors": ["Jane Smith", "John Doe"], "journal": "Nature", "mesh_terms": ["SARS-CoV-2", "Vaccines", "Immunization, Secondary"], "article_types": ["Journal Article", "Research Support, N.I.H., Extramural"], "publication_date": "2024-03-15", "pubmed_url": "https://pubmed.ncbi.nlm.nih.gov/38012345", "pdf_url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10123456/pdf/", "source": "full_text", "chunks": [ { "idx": 0, "text": "...", "tokens": 487 }, { "idx": 1, "text": "...", "tokens": 512 } ] } ``` `pmcid`, `doi`, `pdf_url` may be null. `source` is `"full_text"` when PMC OA NXML was parsed, `"abstract"` when it fell back. #### Data table | Field | Type | Description | |---|---|---| | `pmid` | string | PubMed ID (primary identifier) | | `pmcid` | string? | PubMed Central ID (present for PMC OA papers) | | `doi` | string? | DOI (when provided by PubMed) | | `title` | string | Paper title | | `abstract` | string | Abstract as returned by E-utilities | | `authors` | string\[] | Author display names, order preserved | | `journal` | string | Journal title | | `mesh_terms` | string\[] | MeSH Descriptor headings assigned by NLM | | `article_types` | string\[] | PubMed publication types | | `publication_date` | ISO date | `YYYY-MM-DD` | | `pubmed_url` | string | Link to the PubMed citation | | `pdf_url` | string? | PMC PDF link, when available | | `source` | `"full_text"` | `"abstract"` | Text origin | | `chunks` | Chunk\[] | Token-aware chunks for RAG | | `chunks[].idx` | number | 0-indexed position | | `chunks[].text` | string | Chunk text | | `chunks[].tokens` | number | Token count (≤ 512) | ### 💰 Pricing **$0.02 per paper** (PPR, pay per result). #### How much does it cost to scrape PubMed? | Volume | Estimated cost | |---|---| | 100 papers | **~$2** | | 1,000 papers | **~$20** | | 10,000 papers | **~$200** | | 100,000 papers | **~$2,000** | No subscription. No minimum. You pay only for successful records. ### ⭐ Enjoying PubMed Scraper for RAG?
⭐ ⭐ ⭐ ⭐ ⭐
If this Actor saved you from writing another PubMed XML parser, leave a quick review.
A 5-star rating takes 10 seconds and helps other RAG engineers and biomedical researchers find it. Your feedback also tells us what to build next.
★  Rate this Actor on Apify
### ✨ Tips - **Get an NCBI API key** (free at https://www.ncbi.nlm.nih.gov/account/settings/) for 10 req/s. Turns a 55-min run into roughly 17 min for 1000 papers. - **Use specific `searchTerm` syntax.** PubMed query operators like `[ti]`, `[mh]`, `[au]`, `[dp]` work inside the `searchTerm` field. Example: `cancer[mh] AND immunotherapy[ti] AND 2024[dp]`. - **Filter article types** to cut noise. `["review", "meta_analysis"]` removes editorials, letters, and news pieces. - **`source: "abstract"` is expected for most records.** Only PMC OA papers have full-text NXML. Plan your RAG index accordingly (abstracts are dense and still make good chunks). - **Schedule weekly runs** to keep an embedding index fresh on newly-published biomedical research. - **For large backfills**, split into month-sized windows and run in parallel (separate Actor runs) to stay under the rate limit cleanly. ### ❓ FAQ and limitations #### Is scraping PubMed legal? PubMed's E-utilities API is explicitly designed for programmatic access ([E-utilities docs](https://www.ncbi.nlm.nih.gov/books/NBK25501/)). This Actor respects NCBI's stated rate limits (3 req/s without API key, 10 with) and fetches only publicly-available content. Full-text comes from the PMC Open Access subset, which is licensed for reuse under CC-BY or equivalent. #### What's not in v1? - **Other databases** (bioRxiv, medRxiv, ClinicalTrials.gov, Semantic Scholar). arXiv has its own companion Actor in this portfolio. - **Species / affiliation / journal / language filters** are deferred to a v2 premium tier. - **Citation graph** extraction. - **PDF OCR** for non-PMC papers (they fall back to abstract). - **Section-aware splitting** (Abstract / Methods / Results as separate records). - **MeSH hierarchy expansion** (searching `Cancer` does not auto-include child terms). - **Real-time streaming** or incremental sync. #### Rate limits - **Without API key:** 3 req/s (~55 min for 1000 papers) - **With API key:** 10 req/s (~17 min for 1000 papers) Get a free key at https://www.ncbi.nlm.nih.gov/account/settings/. #### Support Found a bug or want a feature? Use the **Issues** tab on the Actor's page. Custom requirements (other databases, different chunking, section-aware splitting)? Reach out via the Actor's Support link. #### Disclaimer Output metadata is from PubMed's public E-utilities API. Full text, when available, comes from the PMC Open Access archive, which is licensed under CC-BY or equivalent open licenses. Check individual papers' licenses on their PMC page before downstream commercial use. **Built with Apify + Crawlee + TypeScript.** Part of the [actorstack](https://github.com/Devansh-365/actorstack) portfolio. Sister Actor: [arXiv Scraper for RAG](https://apify.com/devanshlive/arxiv-rag-extractor). ### 🔗 Other actors - [arXiv scraper for RAG: papers as chunked JSON](https://apify.com/getascraper/arxiv-rag-extractor) ↗ - turns arXiv preprints into token-aware chunks for RAG pipelines. - [bioRxiv and medRxiv scraper for RAG: chunked JSON](https://apify.com/getascraper/biorxiv-medrxiv-rag-extractor) ↗ - chunks bioRxiv and medRxiv preprints into RAG-ready JSON. - [SEC EDGAR Scraper for RAG: 10-K/10-Q/8-K as JSON](https://apify.com/getascraper/sec-edgar-rag-extractor) ↗ - extracts SEC filings as chunked JSON for financial RAG systems. - [CourtListener RAG extractor](https://apify.com/getascraper/courtlistener-rag-extractor) ↗ - converts US court opinions into chunked JSON for legal RAG pipelines. # Actor input Schema ## `searchTerm` (type: `string`): Enter any search query (such as 'heart disease' or 'SARS-CoV-2 vaccines'). ## `meshTerms` (type: `array`): Add specific Medical Subject Headings if you want to narrow down your medical topic. ## `articleTypes` (type: `array`): Filter by article type (such as clinical trials, reviews, or meta-analyses). Leave empty for all. ## `dateFrom` (type: `string`): Collect papers published on or after this date. ## `dateTo` (type: `string`): Collect papers published on or before this date. ## `maxPapers` (type: `integer`): Set the maximum number of papers you want to save for this run. ## `ncbiApiKey` (type: `string`): Paste your free NCBI API key to speed up collection. Get one on your NCBI settings page. ## Actor input object example ```json { "searchTerm": "SARS-CoV-2 vaccines", "meshTerms": [], "articleTypes": [], "dateFrom": "2024-01-01", "dateTo": "2024-01-15", "maxPapers": 50 } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchTerm": "SARS-CoV-2 vaccines", "meshTerms": [], "articleTypes": [], "dateFrom": "2024-01-01", "dateTo": "2024-01-15", "maxPapers": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("getascraper/pubmed-rag-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchTerm": "SARS-CoV-2 vaccines", "meshTerms": [], "articleTypes": [], "dateFrom": "2024-01-01", "dateTo": "2024-01-15", "maxPapers": 50, } # Run the Actor and wait for it to finish run = client.actor("getascraper/pubmed-rag-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchTerm": "SARS-CoV-2 vaccines", "meshTerms": [], "articleTypes": [], "dateFrom": "2024-01-01", "dateTo": "2024-01-15", "maxPapers": 50 }' | apify call getascraper/pubmed-rag-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,getascraper/pubmed-rag-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Zd6dqQcd4Ikxz2zKK/builds/MVKDBQcVMiEUXtW2r/openapi.json