Evaluation of search-enabled Pre-trained Large Language Models on retrieval tasks for the PubChem Database

Abstract

Databases are indispensable in biological and biomedical research, hosting vast amounts of structured and unstructured data, facilitating the organization, retrieval, and analysis of complex data. Database access, however, remains a manual, tedious, and sometimes overwhelming, task. We investigate in this study the current state of using pre-trained, search-enabled LLMs for data retrieval from biological databases. Equipped with internet search and code generation capabilities, LLMs promise to streamline database access through natural language, expedite search and knowledge retrieval, and provide coherent analytical summaries. As an example database, we focus on evaluating a current search-enabled LLMs (GPT-4o) for retrieval from the PubChem database, a flagship, heavily used database that plays a critical role in biological and biomedical research. As PubChem is an open archival repository, it provides a well-documented programmatic interface that can be exploited through LLM code generation capabilities. We evaluate retrieval tasks for eight common PubChem access protocols that were previously documented. The tasks include identifying interacting genes and proteins, finding drug-like compounds based on structural similarity, retrieving bioactivity data, and locating stereoisomers and isotopomers. We develop a methodology for adopting the protocols into an LLM-prompt, where we supplement the prompt with additional context through iterative prompt refinement as needed. To further evaluate the LLM capabilities, we instruct the LLM to perform the retrieval with and without using programmatic access. We compare the results (referred to as gold and silver answers) when using these retrieval modalities with two traditional retrieval baselines that include running the manual search steps for each reference protocol through the PubChem database web interface, and through the provided PUG (Power-User Gateway) programmatic access. We quantitatively and qualitatively summarize our results, showing that generating programmatic access is more likely to yield the correct answers. We highlight the value and limitations of using current search-based LLMs for database retrieval. We also provide guidance for the future development that can improve the accuracy and reliability of search-based LLMs.