henhao
Sign in
BioinformaticsbioRxiv
Preprint

Spurious model comparisons are widespread in biomedical artificial intelligence

Tianchu ZengiD, Hetu LiiD, Shaoshi ZhangiD, Yan Quan Tan, Fang TianiD, Csaba OrbániD, Lijun AniD, Wanyu CheiD, Jingwen ChengiD, Joanna Su Xian ChongiD, Niousha DehestaniiD, Zijian DongiD, Xin LiiD, Zhizhou Li, Mervyn Jun Rui LimiD, Yi LiniD, Qinrui LingiD, Zijie LingiD, Xi Zhi LowiD, Sina Mansour L.iD, Kwun Kei NgiD, Thuan Tinh NguyeniD, Leon Qi Rong OoiiD, Shreya PandeiD, Xing QianiD, , B.T. Thomas YeoiD

Abstract

Cross-validation is routinely used to compare performance in biomedical artificial intelligence. Standard tests ignore correlation across cross-validation folds, inflating false-positive rates. In a PRISMA-guided meta-analysis of 184 studies (impact factor [≥] 15) across 30 biomedical fields, 97% use invalid tests. Among studies with abstract-level claims supported by invalid tests, 59% rely on a spurious comparison: one that loses significance after correcting for this correlation. On average, regaining significance requires a 43% larger effect. Extending these findings to the broader literature, we estimate that spurious comparisons occur in nearly two in three studies and support abstract-level claims in nearly one in three studies. Simulations confirm false-positive rates paradoxically approach 100% when cross-validation is repeated to improve stability. We introduce SHARP, a redesign of cross-validation, which best balances false-positive control and power among 13 benchmarked tests. These results reveal widespread fragility in biomedical artificial intelligence and offer a practical route to valid comparisons.

PDFDOIData
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
1

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 7 Nov 2026 · from the full text of version 3 (10 Sept 2026) · CC BY 4.0

97% of 184 high-impact biomedical AI studies compare cross-validated models with tests assuming fold independence; in 69% of re-analyzable studies a significant comparison does not survive correction.

Standard paired tests on cross-validation folds treat correlated fold estimates as independent; the paper notes the problem was flagged over two decades ago, but its prevalence and effect on published claims were unmeasured. It quantifies both across 30 biomedical fields and introduces SHARP, a split-half design that estimates fold variance and correlation rather than fixing the correlation (corrected resampled t-test) or sacrificing power (5×2 and empirical tests).

Key results

  1. ShownNearly all included studies used a test that treats cross-validation folds as independent. This did not differ detectably by publication year, field, journal impact factor, journal policy rigor or data/code sharing.

    178 of 184 studies (97%); 95%, 97% and 100% for classical machine learning, deep networks, or both. The naïve t-test and Wilcoxon tests were most common. No detected association with year (permutation p = 0.63), field (p = 0.07), impact factor (p = 0.58), rigor score (p = 0.89) or data/code availability (p = 1.00). Fields with at least 10 studies ranged from 82% (Neuroscience) to 100%. · Invalid statistical tests in 97% of studies; Invalid testing is prevalent regardless of year…

  2. ShownWhen re-tested with a correlation-aware test, most re-analyzable studies had at least one originally significant comparison that lost significance, often by a wide margin rather than just missing the threshold.

    47 of 68 re-analyzable studies (69%). Of 37 studies whose abstract claims rested on invalid tests, 22 (59%) had a spurious supporting comparison. Median per-study largest corrected p = 0.23; median corrected p = 0.17 for abstract-level cases (|z| 1.37 vs 1.96), which the authors equate to needing a 43% larger effect. Single-dataset comparisons were re-tested with the corrected resampled t-test. · Most studies contain a spurious comparison; Re-analysis: loss of significance under a valid…

  3. ShownIn null simulations on four real datasets, the paired t-test and the Wilcoxon test gave inflated false positive rates (FPR). Repeating cross-validation made this worse, with FPR approaching 100%, and sample-level statistics inflated it further.

    Across 420 scenarios: FPR about 19% for one 10-fold run (95% CI entirely above 0.05 in about 64% of scenarios) vs 77% with 30 repeats (inflated in 100% of scenarios). KEGG dataset: sample-level vs fold-averaged FPR 38% vs 24% for one run and 84% vs 81% for 30 repeats. · Tests assuming fold independence poorly control…; Sample-level statistics compound false positive…

  4. ShownSHARP runs K-fold cross-validation separately in two random disjoint halves at each repetition. It controlled false positives as well as the conservative correlation-aware tests, had the highest average power and gave near-nominal interval coverage.

    Over 420 scenarios: the lower bound of the FPR CI was below 5% in nearly all of them. SHARP5-R power was 27.8 percentage points above the 5×2 paired t-test, and it won every pairwise comparison. Mean 95% CI coverage was 96% for SHARP5-R and SHARP10-R. The corrected resampled t-test exceeded 5% FPR in 19% (5-fold) and 44% (10-fold) of scenarios. Under an alternative scheme, SHARP5-R and CorrT5-R had similar power. · SHARP: reliable FPR control through direct…; SHARP matches or exceeds the power of existing…; SHARP provides well-calibrated confidence…

  5. SuggestedExtrapolating the re-analysis rate to every at-risk study, the authors estimate that about two in three studies contain a spurious comparison, and about one in three has an abstract-level claim resting on one.

    (173/184) × (47/68) = 65% of studies; (100/184) × (22/37) = 32% for abstract-level claims. This assumes the 105 at-risk studies that could not be re-analyzed behave like the 68 that were. Generalizing beyond the corpus rests on finding no association with impact factor or rigor score. · Most studies contain a spurious comparison; A serious threat to replicability in biomedical…

Methods, data and reproducibility

PRISMA-guided screening of 2,400 PubMed records (journals with impact factor ≥15, 1 June 2020 to 1 June 2025), with LLM-assisted, human-verified extraction, then re-analysis of originally significant comparisons with a correlation-aware test. The calibration, power and CI coverage of 13 tests were then benchmarked in 420 label-permutation simulation scenarios on four public datasets.

  • Published biomedical AI studies comparing models with cross-validation (PubMed search; journals with impact factor ≥15; 30 fields) · n = 2,400 screened; 184 included; 68 re-analyzed
  • EMNIST Digits grayscale images, ten-class digit classification · n = 240,000 images
  • UK Biobank regional gray-matter volumes (101 regions) used to predict chronological age · human (UK Biobank participants) · n = 36,000 participants
  • Covertype, 54 cartographic features, recast as binary classification (cover type 2 vs rest) · n = 581,012 samples
  • KEGG Metabolic Pathway, 20 graph-topological features, predicting clustering coefficient · n = 53,414 samples
Methods & tools
SHARP (Split-Half Analysis of Repeated Performance), corrected resampled paired t-test (Nadeau-Bengio correction), 5×2 paired t-test, empirical test of differences, DeLong test, PubMed, Journal Citation Reports (Web of Science), Claude Opus 4.1 (LLM-assisted screening), GPT-5 (journal policy scoring), PyCaret, scikit-learn, statsmodels 0.14.5
Data & accessions
https://www.ukbiobank.ac.uk/https://www.kaggle.com/datasets/crawford/emnisthttps://archive.ics.uci.edu/dataset/31/covertypehttps://archive.ics.uci.edu/dataset/220/kegg+metabolic+relation+network+directed
Reagents & environment
statsmodels 0.14.5 for the Poisson regression; PyCaret and scikit-learn for algorithm selection and models (versions not stated). SHARP is described as an open-source, scikit-learn-compatible implementation.
Reproducibility gaps
The code availability statement holds a placeholder (GITHUB_LINK) rather than a URL, and no link to the SHARP implementation is printed. Per-study results (Supplementary Files 1 and 2) and the LLM prompts (Supplementary Methods) are referenced but not part of this text.
Limitations (6)
  • Only 68 of the 173 at-risk studies could be re-analyzed without extra assumptions. The 65% and 32% prevalence figures extrapolate from these to the other 105 studies and to the wider literature. · stated by the authors
  • Re-analysis used the corrected resampled t-test, which does not always control FPR. It added no multiple-comparison corrections and skipped some comparisons (e.g., DeLong-based); the authors argue this makes the estimates conservative. · stated by the authors
  • A comparison that loses significance has weaker evidence than reported. This does not show that the null holds or that a study's broader conclusions are wrong; between-fold correlation affects uncertainty estimates, not point estimates. · stated by the authors
  • SHARP cross-validates within each half, so relative performance can differ from full-dataset estimates when methods scale differently with training size. It cannot be applied retrospectively to published results and does not address fixed-model transportability. · stated by the authors
  • SHARP was benchmarked only in the authors' own label-permutation simulations, with one algorithm per dataset. Under an alternative scheme, the corrected resampled t-test controlled FPR and matched SHARP5-R's power, so the ranking of tests may depend on the simulation design. · noted in this summary
  • The corpus came from a keyword PubMed query (e.g., outperform, benchmark) limited to journals with impact factor ≥15, and it excluded 54 studies reporting only confidence intervals. It may not represent comparisons phrased or published differently. · noted in this summary

Who should read:Read this if you compare models or biomarkers with single or repeated cross-validation and report p-values or confidence intervals, or if you review papers that do.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISTripathi '22Pond '26Clark '24Ahmed '26Metsch '25Khalil '26
56% semantic matchBioinformatics
Satvik Tripathi, Alisha Isabelle Augustin, Farouk Dako+1 more·AI and Ethics·Oct 24, 2022· 8 citations