Tracking claim changes from preprint to publication across 72,644 biomedical studies using large language models

Key results

1. 01

   ShownMost primary abstract claims survived to publication: about 90% were unchanged or only minorly revised, and major revision was uncommon, though its rate varied by field and claim type.

   Of 72,644 pairs: 39.8% unchanged, 50.0% minor, 10.2% substantial/major (incl. 1.7%, n = 1,246, with the primary claim replaced, added or removed); median Jaccard similarity 0.838. Major revision ranged from 7.2% (bioinformatics) to 17.5% (microbiology) and was 5.4% for methodological vs 11.4% to 11.9% for descriptive, associative and mechanistic claims. Claim type kept in 97.7% of 71,055 pairs. · Primary scientific claims usually persist but…; Patterns of revision vary with claim type…; Abstract

2. 02

   ShownWhen the certainty of a claim's wording changed, the published version more often became more cautious than more confident. This held in every large field, and the shift was largest among major revisions.

   Hedging unchanged in 85.6% and non-assessable in 1.8%; more cautious in 8.4% vs more confident in 4.2% of all pairs (exact two-sided sign test on 9,150 shifted pairs, P < 0.001). Among major content changes, 38.5% of assessable claims became more cautious vs 19.8% more confident. Each of the 17 fields having 1,500 or more pairs showed fewer confident than cautious shifts. · Primary scientific claims usually persist but…; Abstract

3. 03

   ShownContent-change labels from Claude Sonnet 4.6 agreed with four human raters nearly as well as the raters agreed with one another. When each abstract was processed blind to its counterpart, the model chose the same or an overlapping primary claim in most cases.

   On a 550-pair subsample enriched for major changes, quadratic-weighted κ was 0.69 vs individual raters and 0.75 vs the mean rating, against 0.76 among raters (range 0.70 to 0.82); model within one ordinal level on 98.7%. Repeat runs agreed exactly on 84.2% to 85.0%. Blind single-abstract extraction (500 pairs) chose the same or overlapping primary claim in 96% (preprint) and 95% (published). · Study design and LLM validation; Claim extraction and classification

4. 04

   ShownMajor revision was more frequent the longer a preprint took to reach publication. Excluding preprints posted within 90 days of publication, which may already include peer review, barely changed the major-revision rate, suggesting such deposits do not account for the overall stability.

   Major revision rose from 7.0% in the fastest interval tertile (median 110 days) to 14.1% in the slowest (median 416 days; chi-squared P < 0.001); a fractional-polynomial model predicted 6.8% at 90 days, 12.3% at 365 and 17.5% at 730. Removing pairs posted within 90 days of publication changed the overall rate from 10.2% to 10.7%. · Patterns of revision vary with claim type…; Statistical analyses

5. 05

   SuggestedIn an exploratory cohort, journal articles linked to a bioRxiv preprint were retracted at a lower rate than articles without an identified preprint.

   Cohort of 399,928 research articles in 47 journals (2021 to 2024): 0.79 vs 5.20 retractions per 10,000 article-years (6 vs 715 events); hazard ratio 0.22 (95% CI 0.10 to 0.50) from a Cox model stratified by journal and publication year; Firth-penalized hazard ratio 0.24 (0.10 to 0.48). · Patterns of revision vary with claim type…; Statistical analyses; Discussion