henhao
Sign in
Scientific Communication & EducationbioRxiv
Preprint

Estimating the replicability of Brazilian biomedical science

The Brazilian Reproducibility Initiative, Olavo B. AmaraliD, Clarissa F. D. CarneiroiD, Kleber NevesiD, Ana Paula Wasilewska‐SampaioiD, Bruna Valério‐GomesiD, Mariana Boechat de AbreuiD, Pedro Batista TaniD, Gabriel Paz Souza MotaiD, Ricardo Netto GoulartiD, Nathalia Raquel de Souza FernandesiD, Jimmy Hayden LinharesiD, A. M. G. IbelliiD, Adriano Defini AndricopuloiD, Adriano SebollelaiD, Adryano Augustto Valladão de Carvalho, Airton Pereira e SilvaiD, Alana Silva Oliveira Souza, Aryelle Almeida LimaiD, Alessandra Mara de SousaiD, Alexander BirbrairiD, Alexandre Urban BorbelyiD, Aline Maria MachadoiD, Alinne C. CostaiD, Aliny Pereira de VasconcelosiD, , Igor L. BaptistaiD

Abstract

Concerns over the replicability and reproducibility of published research have grown in many research fields, but empirical data to inform policies are still scarce. Biomedical research in Brazil expanded rapidly over the last three decades, with no systematic assessment of the replicability of its findings. With this in mind, we set up the Brazilian Reproducibility Initiative, a multicenter replication of published experiments from Brazilian science using three common experimental methods: the MTT assay, the reverse transcription polymerase chain reaction (RT-PCR) and the elevated plus maze (EPM). A total of 56 laboratories performed 143 replications of 56 experiments; of these, 90 replications of 45 experiments were considered valid by an independent committee. Replication rates for these experiments varied between 20 and 44% according to five predefined criteria. In median terms, ratios between group means were 58% lower in replications than in original experiments, while coefficients of variation were 82% higher. Effect size decrease was smaller for MTT experiments, original results with less variability and those considered more challenging to replicate, while t values for replications were positively correlated with researcher predictions about replicability, and negatively correlated with the rate of publications by the original articles last author. Deviations from preregistered protocols were very common in replications, most frequently due to reasons inherent to the experimental model or related to infrastructure and logistics. Our results highlight factors that limit the replicability of results published by researchers in Brazil and suggest ways by which this scenario can be improved.

PDFDOICodeData
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
13
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 5 (4 Feb 2026) · CC BY 4.0

In validated multicenter replications, 20 to 44% of 45 published Brazilian MTT, RT-PCR and elevated plus maze experiments replicated, depending on criterion; replication effects were usually smaller.

Earlier biomedical replication projects covered selected papers in specific fields such as cancer biology, and replicability data usable by a single country's funders and institutions are scarce. Here experiments are sampled by common method from Brazilian publications rather than by field or citation count, and each was assigned to up to three labs blind to the original result, so interlaboratory variation can be separated from original-versus-replication discrepancies.

Key results

  1. ShownDepending on which of five predefined criteria is used, 20 to 44% of validated experiments replicated (the 44% prediction-interval criterion applies to the 39 experiments with multiple replications); a multiverse of analysis choices gives a median replication rate of 25%.

    Primary set (90 valid replications of 45 experiments): original estimate inside the replication 95% prediction interval 17/39 (44%); replication aggregate inside original 95% CI 13/45 (29%); significant same-direction aggregate 9 (20%); half or more replications significant 10 (22%); at least half of labs judged success 19 (42%). Specification curve median 25% (IQR 19-33%). · Replication rates; Concluded experiments; Data analysis

  2. ShownReplication effects were smaller than the originals in nearly all experiments, and original and replication effect sizes correlated only weakly once one extreme outlier was removed.

    41 of 45 replication aggregates (91%) were smaller; median original/replication ratio of ratios 1.71 (MTT 1.51, PCR 1.75, EPM 2.01); median relative differences 80% original vs 8% replication. Pearson r = 0.82 (p = 6.4×10^-12) fell to 0.22 (p = 0.16) without outlier PCR147; Spearman ρ = 0.35 (p = 0.02). · Effect size and variability comparison; Figure legends

  3. ShownReplications were noisier than the originals, most of all for RT-PCR, yet originals differed from their replications more than independent replications differed from each other (exploratory analyses).

    Coefficient of variation lower in the original in 33/45 experiments (73%), median ratio 0.55 (Wilcoxon p = 1.3×10^-5); PCR 0.28 (p = 2.4×10^-4), MTT 0.85 (p = 0.15), EPM 0.74 (p = 0.05). Original-vs-replication differences exceeded between-replication ones, median ratio 1.21 (p = 0.01); EPM 1.55 (p = 0.007), PCR 1.36 (p = 0.19), MTT 1.12 (p = 0.64). · Effect size and variability comparison; Data analysis

  4. SuggestedResearchers' forecasts tracked replication strength, and originals from last authors with high publication rates replicated less well, though many predictors were tested and correlations were weak.

    Aggregate replication t value vs predicted probability of success (seventy forecasting researchers): ρ = 0.46 (p = 0.002); vs last author's publications in the preceding 5 years: ρ = -0.31 (p = 0.04). Effect-size exaggeration rose with original coefficient of variation (ρ = 0.41, p = 0.005) and was lower for MTT (ρ = -0.39, p = 0.008). · Predictors of replication success; Researcher predictions

  5. ShownAlmost every replication deviated from its preregistered protocol, mostly for reasons inherent to the experiment itself (for example, a model system not behaving as anticipated), followed by lab infrastructure and logistics problems.

    Deviations recorded in 95% of replications (MTT 88%, PCR 98%, EPM 100%). Mean deviation rating on a 1 to 5 scale: 1.9 ± 1 by labs vs 2.5 ± 1.1 by the validation committee. The committee excluded 53 of 143 replications, 18 for protocol deviations and 12 for an inadequate experimental unit. · Self-assessment by labs and coordinating team; Concluded experiments

Methods, data and reproducibility

MTT, RT-PCR and elevated plus maze experiments came from Brazilian Web of Science articles (1998-2017). Each went to up to three independent labs, blind to original results, which completed protocols drafted by the coordinating team; protocols were reviewed and preregistered. A committee blind to outcomes validated replications; valid ones were meta-analysed (metafor) and scored on five predefined criteria.

  • Experiments from Brazilian life-science articles in Web of Science (1998 to 2017) using MTT, RT-PCR or elevated plus maze; 317 had protocol data extracted · rodents or cell lines (species not stated) · n = 56 experiments replicated; 45 in primary analysis
  • Replication dataset: 143 replications performed by 56 labs, 90 judged valid · rodents or cell lines (species not stated) · n = 143 replications (90 valid) · 10.7910/DVN/ZJRDIV
  • Researcher survey predicting replication success, effect size and difficulty, plus prediction markets · n = Seventy participants (fifty-seven also in prediction markets) · 10.7910/DVN/2RLSMG
  • Project evaluation forms and semi-structured interviews with participants · n = 121 form responses; 18 interviews
Methods & tools
MTT assay, RT-PCR / RT-qPCR, elevated plus maze, Web of Science, R, metafor, OSF
Code & protocols
https://github.com/BrazilianReproducibilityInitiative/bri-analysishttps://osf.io/4sy2ghttps://osf.io/vs5rp
Data & accessions
https://doi.org/10.7910/DVN/ZJRDIVhttps://doi.org/10.7910/DVN/2RLSMGhttps://osf.io/pwfzehttps://osf.io/vzam6/https://osf.io/9ta45https://osf.io/4tu2v
Reproducibility gaps
R and metafor versions not stated. The analysis protocol was updated after results were available but before analyses were performed, and later deviations are listed at https://osf.io/9hj7t. Compilation errors found after the first preprint version led to data revisions in 26 replications (18%).
Limitations (6)
  • Many replications were underpowered because their coefficients of variation exceeded the originals' (4-fold median difference for PCR). This pushed significance-based rates down and prediction-interval rates up, yet among 35 experiments with ≥80% a posteriori power, rates were still 26 to 38%. · stated by the authors
  • Attrition and exclusion shrank the sample: 21% of planned replications were never done (materials unavailable, or labs leaving or missing deadlines), and validation removed 37% of those performed, leaving only 9 experiments with three valid replications. · stated by the authors
  • Only one comparison per article was replicated (the first statistically significant one), often an initial experiment or a low treatment dose, so a failed replication does not necessarily call an article's main conclusions into question. · stated by the authors
  • The 'one-shot' design, with protocols fixed in advance, likely underestimates replicability compared with 'best-shot' attempts in which labs adjust protocols over repeated tries. · stated by the authors
  • Predictor correlations are weak, many predictors were tested, and replications by overlapping teams are not independent, so these associations may be chance findings. · stated by the authors
  • Experiments were chosen for feasibility (three common assays, commercially available materials, cost cap of BRL 5,000 per replication), so the rates may not carry over to more complex or costly experiments. · noted in this summary

Who should read:Read this if you design replication projects or research policy, or want empirical replication rates and failure modes for routine lab assays such as MTT, RT-PCR and the elevated plus maze.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISCobey '23Freedman '17Grabitz '17Buzbas '22Voelkl '21Buzbas '24
68% semantic matchScientific Communication & Education
K. D. Cobey, S. Ebrahimzadeh, M. Page+4 more·bioRxiv·Sep 18, 2023