SciMuse

SciMuse brings AI into the earliest stage of research: finding and prioritising new research questions. The system builds on a knowledge graph of scientific concepts derived from the literature. For an individual researcher, it analyses their publications, selects related concepts, and uses a large language model (GPT-4) to formulate personalised project ideas and possible collaborations. The practice includes a large-scale human evaluation. More than 100 research group leaders across the natural sciences, social sciences and humanities judged how interesting these AI-generated ideas were. Their judgements then served to predict the interest of new ideas, both through supervised neural networks and through zero-shot ranking with LLMs. AI thus acts as a source of inspiration and a filter, while scientists remain the judges of what is worth pursuing.

SciMuse grew out of the group’s earlier work using semantic networks and machine learning to predict research trends, first in quantum physics. It was motivated by a problem shared across fields: the growth of the scientific literature makes it increasingly hard to identify novel and impactful ideas, especially ones that cross disciplinary boundaries.

The technical basis is open scholarly data. The concept list, over 123,000 concepts, was extracted from about 2.44 million preprints on arXiv, bioRxiv, ChemRxiv and medRxiv. The links between concepts were built from how often those concepts appear together in the titles and abstracts of more than 58 million papers in OpenAlex.

The decisive organisational condition was the institutional setting. The Max Planck Society made it possible to recruit experienced group leaders from very different institutes as evaluators, including Biogeochemistry, Astrophysics and Quantum Optics, as well as Geoanthropology, Demographic Research and Human Development. This gave a genuinely cross-domain test that would be difficult to organise in a single university department.

The practice is also grounded in explicit epistemological reflection. Krenn and philosopher Heather Champion argue that notions such as curiosity, surprise, interest and novelty should be translated into computable objectives for AI in science. SciMuse is an empirical attempt to operationalise one of these notions, scientific interest, through expert judgement.

The main contribution of SciMuse is empirical evidence on how researchers in different fields respond to AI-generated research ideas, and whether that response can be predicted. The expert evaluations created a dataset of more than 4,400 rated ideas, which allowed the team to relate properties of ideas to how interesting researchers found them. The same data showed that interest can be predicted in two ways: by supervised models trained on the human ratings, and by LLMs ranking ideas zero-shot without any such training data. The latter matters where expert data are expensive or unavailable. The work has been presented at the ICML 2024 AI4Science workshop and is available as a preprint.

The practice combines openness with protection of participants. The code and the SciMuse Benchmark are public, and the benchmark lets other research teams test how well their models predict experts’ rankings of research ideas. Because the ideas were personalised to individual researchers, the research questions and expert rankings themselves are kept private for privacy reasons. This also prevents the benchmark from leaking into model training data.

The practice has limitations. Idea generation depends on a proprietary commercial model, which reduces transparency and adds costs. Evaluating newer models on the benchmark is also expensive, and the authors did not run some models more extensively for that reason.

• Preprint PDF: https://arxiv.org/pdf/2405.17044
• ICML 2024 AI4Science workshop version: https://icml.cc/virtual/2024/36773
• Code and SciMuse Benchmark: https://github.com/artificial-scientist-lab/SciMuse