Scholar

Abstract

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

Intro: Research starts by choosing which prior idea to build on

Scientific research is inherently cumulative, with each project building on previous results and ideas. Current AI systems for research mainly accelerate progress on an already well-defined subproblem, where the remaining work is closer to engineering. Deciding which existing idea a new problem should build on is still mostly left to human researchers. An insightful prior paper can turn a vague question into a well-defined problem, for instance by revealing a limitation that motivates a new direction or suggesting an approach worth adapting, thereby shaping and accelerating—catalyzing—subsequent research. Making this choice well is a defining ability of great scientists, and a critical part of what researchers call research taste.

We call such a paper a catalyst paper: one whose ideas did or would have moved the project forward in its early stage. Only a tiny fraction of the millions of papers in the literature qualify, yet each plays a pivotal role in a project. We asked the lead authors, who did the work and know every part of it, to identify them.

Try it yourself! One of the five papers below is one the authors credit as a catalyst for this project. This five-way multiple choice question is much easier than the full task models face, where they have to pick the catalyst papers out of 191K papers.

We have an interactive game here where you find the catalyst paper yourself. It works best on a PC, so please try it there.

Full set on Hugging Face

[](https://ohmyksh.github.io/project/ScholarCatalyst/#)

◀

Core Research Query Subfield specific Query

▶

Query

Identify the catalyst paper for this query

Submit guess Reveal answer

Other catalyst papers for this query

Author rationale

We evaluate finding catalyst papers at the start of research

We ask whether models can make that decision at the start of a project, when researchers do not yet know what to look for. We reflect this early stage setting in our task: given a research question from the outset of a project, a model must retrieve its catalyst papers from a corpus of 191K papers, all published before the source paper was completed, so it can only draw on the literature that existed at the time.

We consider two query types that reflect how an early stage project develops from a broad central research question into particular subfields, with an inspiring paper potentially shaping the next step at either stage.

• Core Research Query asks which prior ideas could help address the project’s central research question.
• Subfield specific Query approaches the same research question from a particular research subfield, asking which prior ideas could help advance the project from that perspective.



ScholarCatalyst asks whether a system can retrieve prior papers that inspire progress on a research project. Such a paper may share little semantic similarity with the question yet offer an insight that advances the project (right), while topically related work may not (left).

How did we build this testbed?

Evaluating this task requires ground truth about which papers catalyzed a project. Yet published papers document completed projects, while the ideation behind them is largely undocumented, and bibliographies rarely identify which works provided nonobvious, critical insights.

Why do we need a new benchmark? Existing scientific retrieval benchmarks measure facet-level similarity, relevance to a stated information need, or citation-intent signals. These signals do not capture whether a paper could have inspired an early-stage project, especially when the eventual paper does not cite it.



Our automated pipeline builds a retrieval corpus and author-review materials, including structured paper data and candidate papers, which authors review through our interface to revise queries, judge candidate relevance, and provide rationales.

Our Automated Construction Pipeline

To collect this data from scratch, we built a scalable pipeline. Given only an arXiv ID, it gathers the paper and its one hop citation graph, drafts an initial query, and assembles a pool of candidate papers. This lets us keep growing the testbed over time.

Collecting Authors’ Firsthand Accounts

We collected this information directly from the authors, reaching authors of award papers at recent conferences as well as through an open call. The authors reviewed every proposed query and candidate paper, validated, revised, or rewrote them, marked the catalyst papers and hard negatives, and explained each relevance judgment.

Inside ScholarCatalyst

Each completed source paper contributes one Core Research Query and one or more Subfield specific Queries. For every query, authors identify catalyst papers that offered useful ideas, distinguish topically related hard negatives, and provide a rationale for each judgment.



Our data also includes something that is rarely public: the authors' own explanations of why a paper helped, what they hoped to gain from it, and why other related papers did not. Such reasoning behind the ideation is almost never documented in published papers.



An example instance from Spurious Rewards: the authors mark the paper that inspired the project, point out similar papers that did not, and explain each judgment in a rationale.

Full set on Hugging Face

Can current retrieval systems identify the catalyst papers?

Our evaluation corpus of 191K computer-science papers is far smaller than the 3M papers on arXiv, among which researchers are able to find work that inspires a project. Retrieval in this corpus is therefore an easier version of the problem researchers actually face. Can current retrieval systems identify the catalyst papers that inspired a project from among the many topically related papers in the corpus?



• Current retrievers miss most of the papers that inspired a project. General purpose dense retrievers perform best, yet place only 39% of author identified inspiration papers in the top 20 for Core Research Queries and 51% for Subfield specific Queries. Scientific document retrievers, BM25, and LateOn all trail them, so training on citation signals alone does not solve the task.
• Agentic search does not improve over embedding retrieval. In our experiments, grep based search performs worst, finding only 8% of gold query and paper pairs compared with 46% for the tool calling agent. We attribute this to candidate coverage, since an agent sees the corpus only through its own search queries and can inspect just a small fraction of it.

Current systems still struggle to find catalyst papers. Agentic search doesn’t beat embedding retrieval, even with the same retrieval tool underneath. Even Fable 5.1 finds only about half of them, despite potentially having seen the finished papers during training.

Analysis

Inspiration is diverse and not captured by similarity

An LLM judge classifies 663 key inspiration rationales under a nine type taxonomy. Authors most often credit a technique they adapted, empirical findings that supported a direction, an idea extended to a new setting, or a limitation that motivated a new approach. A single paper often inspires through several mechanisms at once, with more than one type applying to 55.8% of rationales.

Inspiration is diverse, ranging from a reusable method to supporting evidence or a new way of framing a problem. Percentages sum to more than 100% because a rationale can receive multiple types.

Topical similarity and citation status do not capture inspiration

Hard negatives are at least as similar to the query as catalyst papers under both lexical and embedding similarity, so topical similarity cannot single out the papers that inspired a project. Citation status is also an incomplete signal. For Subfield specific Queries, 43.6% of catalyst papers are never cited by the source paper. Even among the catalyst papers that are cited, a citation intent classifier assigns a uses or extends label to only 46.2% for Core Research Queries and 34.0% for Subfield specific Queries, so being cited does not mean a paper was built upon.

Recognition depends on candidate coverage

To separate retrieval coverage from recognition performance, we apply LLM based reranking to the top 20 candidates from Gemini Embedding 2 on a subset of 100 queries. We compare reranking with and without gold injection, which replaces negative candidates with missing catalyst papers while keeping the pool size fixed. Stronger rerankers gain more in Recall@5 from gold injection. The widening gap suggests that candidate coverage limits what a stronger reranker can deliver, since its ability to pick out useful papers only matters when those papers enter the pool. Better retrieval should therefore yield larger gains as rerankers improve.

Stronger rerankers gain more from gold injection. Recall@5 after reranking the retriever's top 20, with and without missing gold papers swapped in. Claude Fable rises from roughly 27% to 50%.

Read the full paper for more experiments and analysis →

Outro

Recent work on reasoning-intensive retrieval evaluates relevance relations that require reasoning to recognize. ScholarCatalyst pushes further, requiring judgments of how prior ideas could advance a research project.

We see two milestones ahead. The first is expert-level retrieval, matching a domain expert's judgment within their own field. The second is retrieval beyond any individual expert: being able to exercise that level of intuition across a breadth of fields no single researcher can follow.

This breadth may be where such systems help most, since nearly half of the papers our authors credit (49.5%) come from outside their project's subject area, and 43.6% of subfield-specific positives were never cited in the source paper. Researchers may therefore be limited more by what they have seen than by how well they judge it. The full literature makes this gap far larger than our benchmark shows: Semantic Scholar alone indexes over 225M papers, compared with 191K in our corpus.

What stands between current systems and both milestones is an expert-level sense of which ideas matter, and we hope ScholarCatalyst offers a starting point for developing it. If undiscovered public knowledge (1986) is as common as our results imply, models capable of expert judgment across fields could unlock a large backlog of progress hidden in plain sight.

The ScholarCatalyst benchmark would not have been possible without the researchers who shared how their discoveries came about, and we are deeply grateful for their invaluable contributions.

Benchmark Contributors

Their firsthand knowledge is the foundation of this benchmark, and we thank them for generously dedicating their time and expertise. Contributors are listed in alphabetical order.

Adir Dayan, Ajay Sridhar, Alberta Longhini, Alberto Rota, Alexandru Oarga, Ali Hatamizadeh, Alireza Mousavi-Hosseini, Angana Borah, Anjiang Wei, Atharva Kulkarni, Bhuvan Sachdeva, Bo Liu, Bryan Sangwoo Kim, Burouj Armgaan, Camila Blank, Caroline Choi, Changho Shin, Changyeon Kim, Chaofan Lin, Chengyang He, Chuhan Li, David Anugraha, Dayoon Ko, Dilara Soylu, Dongjin Kang, Dongyang Fan, Ekaterina Kochetkova, Eunkyu Park, Fan Nie, Fanhu Zeng, Fatemeh Pesaran Zadeh, Francesco De Santis, Gaotang Li, Giung Nam, Guancheng Zhou, Guangnian Wan, Guijin Son, Gul Sena Altıntas, Gyuhyeon Seo, Han Luo, Hanane Moussa, Hangoo Kang, Haochen Tian, Haoyu Zhang, Haozhe Chen, Haozhe Wang, Himanshu Gupta, Ho Sy Tuyen, Hoang Pham, Houjun Liu, Hwan Chang, Hyeong Kyu Choi, Hyunji Lee, Hyunjun Lee, Ifdita Hasan Orney, Ilia Sucholutsky, Insu Lee, Itay Itzhak, Jaewoo Ahn, Jarek Liesen, Jason S Lucas, Jen-Yen Chang, Jiaju Ma, Jiaqi Liu, Jiayin Zhu, Jingzhe Shi, Jinheon Baek, Jinkun Hao, Jinwoo Lim, Jiwoo Chung, Johannes Kirmayr, Johnny Tian-Zheng Wei, Jongseo Lee, Joonghyuk Shin, Jubayer Ibn Hamid, Junchi Yu, Jungwoo Kim, Junyoung Lim, Kallol Saha, Kevin Qinghong Lin, Kodai Kawamura, Korneel Van den Berghe, Kunwoong Kim, Letian Fu, Linus Kreitner, Lucy Xiaoyang Shi, Lynnette Hui Xian Ng, Madhav Kanda, Mahyar Ghazanfari, Marcel Torne, Marco Bertolini, Marco Fumero, Maria Parelli, Marie Brockschmidt, Markus Gross, Mert Cemri, Michael Rizvi-Martel, Ming Hu, Mingtong Zhang, Mingyu Kim, Minki Kang, Minkyu Kim, Minseo Kim, Na Min An, Nathaniel L. Diamant, Niels Mündler, Nilesh Jain, Nilesh Prasad Pandey, Omin Kwon, Peng Xia, Pengcheng Wang, Pratik Sachdeva, Qingyang Zhang, Qizheng Zhang, Rahul Sharma, Raj Ghugare, Richard Cornelius Suwandi, Rohit Jena, Rosen Ting-Ying Yu, Roshen Sanjay Nair, Roussel Desmond Nzoyem, Rulin Shao, Runhan Huang, Sanghyun Jo, Seohong Park, Seokwon Song, Seonghyeon Ye, Seungone Kim, Shayan Talaei, Shihao Wang, Shiyin Jiang, Sijia Liu, Sijie Zhao, Siqi Zhu, Taehoon Yoon, Tanvir Ahmed Sijan, Tianhe Wu, Tongtian Zhu, Truong Buu Phan, Valentin Hofmann, Valeria Ruscio, Vasily Ilin, Wei Chen, Weiheng Liu, Wenbin Ouyang, Wenjie Zhu, Woomin Song, Woongyeong Yeo, Xiang Li, Xiaotian Liu, Xuan Luo, Xuanming Zhang, Xuying Ning, Yancheng Zhang, Yang Li, Yangcen Liu, Yanggan Gu, Yangtian Zhang, Yanjiang Guo, Yaozhong Shi, Yasaman Haghighi, Yichao Cai, Yifan Zhang, Yoonho Lee, Yoonkyo Jung, Yoonsang Lee, Yuante Li, Yuejiang Liu, Yujia Zheng, Yumin Choi, Yusuf Dalva, Zeyuan Chen, Zhaoyang Liu, Zheng Huang, Zhenhao Chen, Zhongxing Xu, Zijian Feng, Zixuan Hu

Citation

BibTeXCopy

@article{kim2026scholarcatalyst,
    title   = {ScholarCatalyst: A Benchmark for Retrieving Papers that Inspire New Research},
    author  = {Kim, Sohyeon and Lee, Yoonho and Liu, Bo and Ko, Dayoon and Shao, Rulin and Kim, Seungone and Neubig, Graham and Koh, Pang Wei and Chowdhery, Aakanksha and Asai, Akari and Khattab, Omar and Choi, Yejin and Kim, Gunhee and Finn, Chelsea},
    journal = {arXiv preprint arXiv:2610.02202},
    year    = {2026}
}