henhaobio
Sign in

Open Research AfricaScientific Communication & EducationPeer-reviewed journal article

Creating a biomedical knowledge base by addressing GPT inaccurate responses and benchmarking context

Shelby S. DarnelliD, Rupert W. OveralliD, Andrea GuarracinoiD, Vicenza Colonna, Flavia VillaniiD, Erik P. GarrisoniD, Arun IsaaciD, Priscilla Muli, Muriithi Frederick Muriuki, Alexander Kabui, Munyoki KilyungiiD, Felix LissoiD, Adrian Kibet, Brian Muhia, Harm NijveeniD, Joep de LigtiD, Siamak YousefiiD, David George AshbrookiD, P.-S. HuangiD, G. Edward SuhiD, Muhammad UmariD, Christopher BatteniD, Hao CheniD, Śaunak SeniD, Robert W. WilliamsiD, Pjotr PrinsiD

PDFDOIPreprintCode
In-depth full-text digest available · Jump to digest ↓

Abstract

We created GNQA, a generative pre-trained transformer (GPT) knowledge base driven by a performant retrieval augmented generation (RAG) with a focus on aging, dementia, Alzheimers and diabetes. We uploaded a corpus of three thousand peer reviewed publications on these topics into the RAG. To address concerns about inaccurate responses and GPT hallucinations, we implemented a context provenance tracking mechanism that enables researchers to validate responses against the original material and to get references to the original papers. To assess the effectiveness of contextual information we collected evaluations and feedback from both domain expert users and citizen scientists on the relevance of GPT responses. A key innovation of our study is automated evaluation by way of a RAG assessment system (RAGAS). RAGAS combines human expert assessment with AI-driven evaluation to measure the effectiveness of RAG systems. When evaluating the responses to their questions, human respondents give a "thumbs-up" 76% of the time. Meanwhile, RAGAS scores 90% on answer relevance on questions posed by experts. And when GPT-generates questions, RAGAS scores 74% on answer relevance. With RAGAS we created a benchmark that can be used to continuously assess the performance of our knowledge base. Full GNQA functionality is embedded in the free GeneNetwork.org web service, an open-source system containing over 25 years of experimental data on model organisms and human. The code developed for this study is published under a free and open-source software license at https://git.genenetwork.org/gn-ai/tree/README.md.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (21 Oct 2024) · CC BY 4.0

GNQA, a retrieval-augmented GPT question-answering system over 3000 GeneNetwork, aging and diabetes papers, cites source paragraphs; RAGAS rated answer relevance above human thumbs-up ratings (89% vs 76%).

Researchers cannot keep pace with the literature even on single topics such as diabetes or Alzheimer's disease, and GPT answers can contain hallucinated facts and fabricated references. GNQA links each answer to the specific passages and papers it retrieved, which the authors contrast with OpenAI's publisher-level citations, and it pairs automated RAGAS scoring with ratings from expert and lay users, which they say most recent biology LLM studies lacked.

Key results

  1. ShownGNQA answers questions from a 3000-paper corpus. With each answer it returns the titles of the source papers, the roughly 100-word passages it used (merged when several come from one paper) and PubMed abstracts and links. It is embedded in the GeneNetwork.org search page.

    Corpus: one thousand papers mentioning GeneNetwork, one thousand on aging and one thousand on diabetes. PDFs are parsed with GROBID into sentence-respecting chunks of about 100 words, stored in PostgreSQL, retrieved by HNSW search and ranked by GPT yes/no log probabilities. Screenshots show this for a question on non-coding RNAs in diabetes. · A. GNQA provides a generative pre-trained…; B. Document Ingestion; C. Retrieval Augmented Generation (RAG); Figure legends

  2. ShownOn volunteer questions, RAGAS (an automated, GPT-based evaluation) judged answers mostly faithful to the retrieved context and relevant to the question. The retrieved context itself scored low on relevance.

    52 expert and 61 citizen-scientist questions, with each dataset scored three times. Averages: faithfulness 87%, context utilization 81%, answer relevance 89%, context relevance below 24%. · Answer relevance; C. Performance rated by domain experts and…; D. RAGAS; Tables

  3. SuggestedDomain experts' questions got better RAGAS scores than citizen scientists' questions. Focused aging and diabetes questions did better than general GeneNetwork questions, especially on answer relevance.

    Overall RAGAS score 74% (experts) vs 67% (citizen scientists). For experts, context relevance was higher by 19% and faithfulness by 6%. Answer relevance was 88% (experts) vs 90% (citizen scientists). Raters: eleven domain experts and ten citizen scientists. · C. Performance rated by domain experts and…; F. Volunteers; Tables

  4. SuggestedSimple thumbs-up/down ratings gave an overall score close to the automated one (76% vs 70%), but on answer relevance RAGAS was more generous than the human raters (89% vs 76%). The authors conclude that RAGAS cannot completely replace human assessment.

    Thumbs-up rate 76% vs RAGAS overall 70% (mean of the expert and citizen-scientist overall scores). RAGAS answer relevance averaged 89% vs 76% for the human thumbs up/down rating. · Abstract; C. Performance rated by domain experts and…; Discussion

  5. SuggestedQuestions generated by GPT-4o scored below human questions. Close to half of the GPT-4o questions about GeneNetwork went unanswered (75% at the expert level), which the authors attribute to GPT splitting the name into 'gene network'.

    120 GPT-4o questions, 20 per topic and expertise level. Aging and diabetes questions got responses. 75% of expert-level GeneNetwork questions did not (only five of 20 answered), nor did close to half of all GeneNetwork ones. Citizen level: humans higher by 16%, 8% and 2% (answer relevance, context utilization, overall); expert level: by 10%. RAGAS answer relevance on GPT questions: 74%. · Abstract; D. Performance rated by RAGAS using…; Figure legends; Tables

Methods, data and reproducibility

A retrieval-augmented generation (RAG) pipeline. PDFs are parsed with GROBID into chunks of about 100 words, retrieved by HNSW similarity search, reranked by GPT-3.5/4/4o acting as a zero-shot yes/no classifier, and answered by GPT with titled references. Answers to volunteer questions were rated thumbs up/down and scored with RAGAS; answers to GPT-4o-generated questions were scored with RAGAS only.

  • Curated PDF corpus: papers mentioning GeneNetwork.org, plus papers on the genetics/genomics of aging and of diabetes · n = 3000 publications (one thousand per topic)
  • Unique questions posed by volunteer domain experts and citizen scientists, rated thumbs up/down and scored with RAGAS · n = 52 + 61 questions from eleven experts and ten citizen scientists
  • Questions generated by GPT-4o from simulated citizen-scientist and domain-expert perspectives on GeneNetwork, aging and diabetes · n = 120 questions
Methods & tools
GNQA, GeneNetwork.org, OpenAI GPT-3.5/GPT-4/GPT-4o, FahamuAI digest, GROBID, PostgreSQL, HNSW, RAGAS, PubMed API, Flask
Reproducibility gaps
Relies on proprietary OpenAI GPT and on FahamuAI's RAG service, a company founded by two of the authors. The text does not list the 3000 corpus papers. Only example RAGAS-ready datasets are said to be in the code repository; all questions are listed in the appendix.
Limitations (6)
  • Experts found the corpus too narrow. They expected broader general genetics knowledge (for example textbooks) and noted that important tool references were missing and some included ones were outdated. · stated by the authors
  • RAGAS rated answers more favourably than humans did (answer relevance 89% vs 76% thumbs up), so it cannot completely replace human assessment. · stated by the authors
  • Using GPT to generate test questions may introduce bias, because the questions come from the same GPT mechanism being evaluated. · stated by the authors
  • The system depends on closed, non-FAIR components (OpenAI GPT and FahamuAI's RAG service). Comparison with open-source LLMs and RAG engines is left to future work. · stated by the authors
  • The human evaluation is small (eleven domain experts and ten citizen scientists; only 6 expert questions on aging), with no statistical testing. The RAGAS metrics used do not check correctness against reference answers. · noted in this summary
  • Scores for GPT-generated questions leave out unanswered questions without a penalty. Some figures differ between sections (expert answer relevance 90% in the abstract vs 88% in the Results; unanswered GPT GeneNetwork questions given as 75% and as 'close to half'). · noted in this summary

Read this if you are building or evaluating a retrieval-augmented LLM over biomedical literature and want an example of passage-level provenance, with RAGAS scores compared against ratings from experts and lay users.

Citations
5
FWCI
1.02
Field percentile
top 22%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISPalem '26Mohammadi '25Bianchi '26Chen '23Yip '25Krithara '22
64% semantic matchBioinformatics
Revanth Reddy Palem, Hao Chen, Zongliang Yue·bioRxiv·May 29, 2026