arxiv

1---2name: arxiv3description: "Search arXiv papers by keyword, author, category, or ID."4version: 1.0.05author: Hermes Agent6license: MIT7platforms: [linux, macos, windows]8metadata:9  hermes:10    tags: [Research, Arxiv, Papers, Academic, Science, API]11    related_skills: [pdf]12---13 14# arXiv Research15 16Search and retrieve academic papers from arXiv via their free REST API. No API key, no dependencies — just curl.17 18## Quick Reference19 20| Action | Command |21|--------|---------|22| Search papers | `curl "https://export.arxiv.org/api/query?search_query=all:QUERY&max_results=5"` |23| Get specific paper | `curl "https://export.arxiv.org/api/query?id_list=2402.03300"` |24| Read abstract (web) | `web_extract(urls=["https://arxiv.org/abs/2402.03300"])` |25| Read full paper (PDF) | `web_extract(urls=["https://arxiv.org/pdf/2402.03300"])` |26 27## Searching Papers28 29The API returns Atom XML. Parse with `grep`/`sed` or pipe through `python` for clean output.30 31### Basic search32 33```bash34curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5"35```36 37### Clean output (parse XML to readable format)38 39```bash40curl -s "https://export.arxiv.org/api/query?search_query=all:GRPO+reinforcement+learning&max_results=5&sortBy=submittedDate&sortOrder=descending" | python -c "41import sys, xml.etree.ElementTree as ET42ns = {'a': 'http://www.w3.org/2005/Atom'}43root = ET.parse(sys.stdin).getroot()44for i, entry in enumerate(root.findall('a:entry', ns)):45    title = entry.find('a:title', ns).text.strip().replace('\n', ' ')46    arxiv_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]47    published = entry.find('a:published', ns).text[:10]48    authors = ', '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))49    summary = entry.find('a:summary', ns).text.strip()[:200]50    cats = ', '.join(c.get('term') for c in entry.findall('a:category', ns))51    print(f'{i+1}. [{arxiv_id}] {title}')52    print(f'   Authors: {authors}')53    print(f'   Published: {published} | Categories: {cats}')54    print(f'   Abstract: {summary}...')55    print(f'   PDF: https://arxiv.org/pdf/{arxiv_id}')56    print()57"58```59 60## Search Query Syntax61 62| Prefix | Searches | Example |63|--------|----------|---------|64| `all:` | All fields | `all:transformer+attention` |65| `ti:` | Title | `ti:large+language+models` |66| `au:` | Author | `au:vaswani` |67| `abs:` | Abstract | `abs:reinforcement+learning` |68| `cat:` | Category | `cat:cs.AI` |69| `co:` | Comment | `co:accepted+NeurIPS` |70 71### Boolean operators72 73```74# AND (default when using +)75search_query=all:transformer+attention76 77# OR78search_query=all:GPT+OR+all:BERT79 80# AND NOT81search_query=all:language+model+ANDNOT+all:vision82 83# Exact phrase84search_query=ti:"chain+of+thought"85 86# Combined87search_query=au:hinton+AND+cat:cs.LG88```89 90## Sort and Pagination91 92| Parameter | Options |93|-----------|---------|94| `sortBy` | `relevance`, `lastUpdatedDate`, `submittedDate` |95| `sortOrder` | `ascending`, `descending` |96| `start` | Result offset (0-based) |97| `max_results` | Number of results (default 10, max 30000) |98 99```bash100# Latest 10 papers in cs.AI101curl -s "https://export.arxiv.org/api/query?search_query=cat:cs.AI&sortBy=submittedDate&sortOrder=descending&max_results=10"102```103 104## Fetching Specific Papers105 106```bash107# By arXiv ID108curl -s "https://export.arxiv.org/api/query?id_list=2402.03300"109 110# Multiple papers111curl -s "https://export.arxiv.org/api/query?id_list=2402.03300,2401.12345,2403.00001"112```113 114## BibTeX Generation115 116After fetching metadata for a paper, generate a BibTeX entry:117 118{% raw %}119```bash120curl -s "https://export.arxiv.org/api/query?id_list=1706.03762" | python -c "121import sys, xml.etree.ElementTree as ET122ns = {'a': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}123root = ET.parse(sys.stdin).getroot()124entry = root.find('a:entry', ns)125if entry is None: sys.exit('Paper not found')126title = entry.find('a:title', ns).text.strip().replace('\n', ' ')127authors = ' and '.join(a.find('a:name', ns).text for a in entry.findall('a:author', ns))128year = entry.find('a:published', ns).text[:4]129raw_id = entry.find('a:id', ns).text.strip().split('/abs/')[-1]130cat = entry.find('arxiv:primary_category', ns)131primary = cat.get('term') if cat is not None else 'cs.LG'132last_name = entry.find('a:author', ns).find('a:name', ns).text.split()[-1]133print(f'@article{{{last_name}{year}_{raw_id.replace(\".\", \"\")},')134print(f'  title     = {{{title}}},')135print(f'  author    = {{{authors}}},')136print(f'  year      = {{{year}}},')137print(f'  eprint    = {{{raw_id}}},')138print(f'  archivePrefix = {{arXiv}},')139print(f'  primaryClass  = {{{primary}}},')140print(f'  url       = {{https://arxiv.org/abs/{raw_id}}}')141print('}')142"143```144{% endraw %}145 146## Reading Paper Content147 148After finding a paper, read it:149 150```151# Abstract page (fast, metadata + abstract)152web_extract(urls=["https://arxiv.org/abs/2402.03300"])153 154# Full paper (PDF → markdown via Firecrawl)155web_extract(urls=["https://arxiv.org/pdf/2402.03300"])156```157 158For local PDF processing, see the `ocr-and-documents` skill.159 160## Common Categories161 162| Category | Field |163|----------|-------|164| `cs.AI` | Artificial Intelligence |165| `cs.CL` | Computation and Language (NLP) |166| `cs.CV` | Computer Vision |167| `cs.LG` | Machine Learning |168| `cs.CR` | Cryptography and Security |169| `stat.ML` | Machine Learning (Statistics) |170| `math.OC` | Optimization and Control |171| `physics.comp-ph` | Computational Physics |172 173Full list: https://arxiv.org/category_taxonomy174 175## Helper Script176 177The `scripts/search_arxiv.py` script handles XML parsing and provides clean output:178 179```bash180python scripts/search_arxiv.py "GRPO reinforcement learning"181python scripts/search_arxiv.py "transformer attention" --max 10 --sort date182python scripts/search_arxiv.py --author "Yann LeCun" --max 5183python scripts/search_arxiv.py --category cs.AI --sort date184python scripts/search_arxiv.py --id 2402.03300185python scripts/search_arxiv.py --id 2402.03300,2401.12345186```187 188No dependencies — uses only Python stdlib.189 190---191 192## Semantic Scholar (Citations, Related Papers, Author Profiles)193 194arXiv doesn't provide citation data or recommendations. Use the **Semantic Scholar API** for that — free, no key needed for basic use (1 req/sec), returns JSON.195 196### Get paper details + citations197 198```bash199# By arXiv ID200curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300?fields=title,authors,citationCount,referenceCount,influentialCitationCount,year,abstract" | python -m json.tool201 202# By Semantic Scholar paper ID or DOI203curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1234/example?fields=title,citationCount"204```205 206### Get citations OF a paper (who cited it)207 208```bash209curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/citations?fields=title,authors,year,citationCount&limit=10" | python -m json.tool210```211 212### Get references FROM a paper (what it cites)213 214```bash215curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:2402.03300/references?fields=title,authors,year,citationCount&limit=10" | python -m json.tool216```217 218### Search papers (alternative to arXiv search, returns JSON)219 220```bash221curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=GRPO+reinforcement+learning&limit=5&fields=title,authors,year,citationCount,externalIds" | python -m json.tool222```223 224### Get paper recommendations225 226```bash227curl -s -X POST "https://api.semanticscholar.org/recommendations/v1/papers/" \228  -H "Content-Type: application/json" \229  -d '{"positivePaperIds": ["arXiv:2402.03300"], "negativePaperIds": []}' | python -m json.tool230```231 232### Author profile233 234```bash235curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=Yann+LeCun&fields=name,hIndex,citationCount,paperCount" | python -m json.tool236```237 238### Useful Semantic Scholar fields239 240`title`, `authors`, `year`, `abstract`, `citationCount`, `referenceCount`, `influentialCitationCount`, `isOpenAccess`, `openAccessPdf`, `fieldsOfStudy`, `publicationVenue`, `externalIds` (contains arXiv ID, DOI, etc.)241 242---243 244## Complete Research Workflow245 2461. **Discover**: `python scripts/search_arxiv.py "your topic" --sort date --max 10`2472. **Assess impact**: `curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID?fields=citationCount,influentialCitationCount"`2483. **Read abstract**: `web_extract(urls=["https://arxiv.org/abs/ID"])`2494. **Read full paper**: `web_extract(urls=["https://arxiv.org/pdf/ID"])`2505. **Find related work**: `curl -s "https://api.semanticscholar.org/graph/v1/paper/arXiv:ID/references?fields=title,citationCount&limit=20"`2516. **Get recommendations**: POST to Semantic Scholar recommendations endpoint2527. **Track authors**: `curl -s "https://api.semanticscholar.org/graph/v1/author/search?query=NAME"`253 254## Rate Limits255 256| API | Rate | Auth |257|-----|------|------|258| arXiv | ~1 req / 3 seconds | None needed |259| Semantic Scholar | 1 req / second | None (100/sec with API key) |260 261## Notes262 263- arXiv returns Atom XML — use the helper script or parsing snippet for clean output264- Semantic Scholar returns JSON — pipe through `python -m json.tool` for readability265- arXiv IDs: old format (`hep-th/0601001`) vs new (`2402.03300`)266- PDF: `https://arxiv.org/pdf/{id}` — Abstract: `https://arxiv.org/abs/{id}`267- HTML (when available): `https://arxiv.org/html/{id}`268- For local PDF processing, see the `ocr-and-documents` skill269 270## ID Versioning271 272- `arxiv.org/abs/1706.03762` always resolves to the **latest** version273- `arxiv.org/abs/1706.03762v1` points to a **specific** immutable version274- When generating citations, preserve the version suffix you actually read to prevent citation drift (a later version may substantially change content)275- The API `<id>` field returns the versioned URL (e.g., `http://arxiv.org/abs/1706.03762v7`)276 277## Withdrawn Papers278 279Papers can be withdrawn after submission. When this happens:280- The `<summary>` field contains a withdrawal notice (look for "withdrawn" or "retracted")281- Metadata fields may be incomplete282- Always check the summary before treating a result as a valid paper283