searching-scientific-literature

kthorn's avatarfrom kthorn

PubMed search with keyword optimization, result parsing, and metadata extraction

⭐2stars🔀0forks📁View on GitHub🕐Updated Oct 22, 2025⬇️1downloads

When & Why to Use This Skill

This Claude skill automates the discovery and extraction of scientific literature from PubMed. It streamlines the research process by optimizing search queries with Boolean logic, managing API rate limits, and extracting comprehensive metadata—including PMIDs, DOIs, and abstracts—to build structured datasets for academic review.

Use Cases

  • Literature Review: Automatically gather and organize a foundational set of papers when starting a new research project or thesis.
  • Targeted Search: Execute complex queries using specific field tags (Title/Abstract, Author, Journal) to find precise information on compounds, diseases, or methodologies.
  • Metadata Extraction: Parse and structure publication details into standardized JSON formats for integration into reference managers or research databases.
  • Systematic Screening: Build initial paper lists for systematic reviews by casting a wide net with optimized synonyms and keyword variations.
nameSearching Scientific Literature
descriptionPubMed search with keyword optimization, result parsing, and metadata extraction
when_to_useWhen starting literature search. When user asks about papers, publications, studies. When need to find scientific articles. When building initial paper list for research question.
version1.0.0

Searching Scientific Literature

Overview

Search PubMed for scientific literature using optimized queries. Extract metadata and prepare papers for relevance evaluation.

Core principle: Cast a wide enough net to find relevant papers, but use targeted keywords to keep results manageable.

When to Use

Use this skill when:

  • Starting a new research question
  • User asks "find papers about..."
  • Need initial paper set for evaluation
  • Searching for specific methods, compounds, diseases, techniques

Search Strategy

1. Parse User Query

Extract:

  • Keywords: Main concepts (e.g., "BTK inhibitor", "selectivity", "kinase")
  • Data types: What user needs (IC50 values, methods, structures, results)
  • Constraints: Date ranges, specific journals, author names
  • Synonyms: Alternative terms (e.g., "Bruton's tyrosine kinase" = "BTK")

2. Construct PubMed Query

Boolean operators:

  • AND - narrow results (must have both terms)
  • OR - broaden results (either term)
  • NOT - exclude terms

Example queries:

"BTK inhibitor"[Title/Abstract] AND selectivity[Title/Abstract]

("kinase inhibitor" OR "protein kinase") AND (selectivity OR "off-target")

"ibrutinib"[Title/Abstract] AND ("IC50" OR "inhibitory concentration")

Field tags:

  • [Title/Abstract] - search title and abstract only
  • [Title] - title only (more precise)
  • [Author] - specific author
  • [Journal] - specific journal
  • [Date] - date range

API endpoint:

https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?\
db=pubmed&\
term=YOUR_QUERY&\
retmax=100&\
retmode=json&\
sort=relevance

Parameters:

  • db=pubmed - search PubMed database
  • term= - your query (URL encode spaces and special chars)
  • retmax=100 - max results (start with 100)
  • retmode=json - return JSON
  • sort=relevance - most relevant first (or pub_date for newest)

Example bash:

curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=BTK+inhibitor+selectivity&retmax=100&retmode=json&sort=relevance"

Response format:

{
  "esearchresult": {
    "count": "156",
    "retmax": "100",
    "idlist": ["12345678", "87654321", ...]
  }
}

4. Fetch Paper Metadata

API endpoint:

https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?\
db=pubmed&\
id=12345678,87654321&\
retmode=json

Extract from response:

  • Title
  • Authors (list)
  • Journal name
  • Publication date
  • Abstract (via separate efetch call or use esummary)
  • PMID
  • DOI (if available in articleids)

Getting DOI from PMID:

"articleids": [
  {"idtype": "pubmed", "value": "12345678"},
  {"idtype": "doi", "value": "10.1234/example.2023"}
]

If DOI missing:

  • Use PMID as fallback identifier
  • Try to resolve DOI via PubMed Central or publisher APIs later

Output Format

Create list of paper objects:

[
  {
    "pmid": "12345678",
    "doi": "10.1234/example.2023",
    "title": "Selective BTK inhibitors for autoimmune diseases",
    "authors": ["Smith J", "Doe A", "Johnson B"],
    "journal": "Nature Chemical Biology",
    "year": "2023",
    "abstract": "We developed a series of...",
    "source": "pubmed_search"
  }
]

Error Handling

Rate limits (CRITICAL - shared across all processes/subagents):

  • No API key: 3 requests/second (official limit)
  • With API key: 10 requests/second
  • Single agent/script: Use 500ms delays (2 req/sec, safe margin)
    • 350ms is theoretically sufficient but causes ~20% HTTP 429 errors in practice
  • Multiple parallel subagents: Use longer delays to share capacity
    • 2 parallel: 1 second each (2 total req/sec)
    • 3 parallel: 1.5 seconds each (2 total req/sec)
    • 5 parallel: 2.5 seconds each (2 total req/sec)
    • Formula: delay_seconds = (num_parallel / rate_limit) + safety_margin
  • If you get HTTP 429 errors: Wait 5 seconds, resume with doubled delays

Empty results:

  • Try broader terms
  • Remove field tags
  • Check for typos
  • Use OR to add synonyms

Too many results (>500):

  • Add more specific terms
  • Use field tags to narrow
  • Add date constraints
  • Consider splitting into sub-queries

Integration with Other Skills

After search completes:

  1. Save results to research folder as initial-search-results.json
  2. For each paper, call evaluating-paper-relevance skill
  3. Track in papers-reviewed.json (use DOI as key, fallback to PMID)

Quick Reference

Task Command
Search PubMed curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=QUERY&retmax=100&retmode=json"
Get metadata curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=PMID1,PMID2&retmode=json"
URL encode query Replace spaces with +, special chars with %XX
Narrow results Use AND, add field tags, more specific terms
Broaden results Use OR, remove field tags, add synonyms

Common Mistakes

Too narrow: Only 5 results → Use OR, remove constraints Too broad: 5000 results → Add AND terms, use field tags Missing abstracts: Use efetch instead of esummary for full abstract text DOI not found: Many older papers lack DOI - use PMID as fallback Rate limiting: Add 500ms delays (single agent) or longer (parallel subagents sharing rate limit)

Next Steps

After completing search:

  • Announce: "Found N papers matching query"
  • Begin evaluation using skills/research/evaluating-paper-relevance
  • Update user with progress as papers are screened