henhao
Sign in
NeurosciencebioRxiv
Preprint

Towards Reproducible Brain-Wide Association Studies

Scott A. MarekiD, Brenden Tervo‐ClemmensiD, Finnegan J. CalabroiD, David F. MonteziD, Benjamin P. KayiD, Alexander Samir HatoumiD, Meghan Rose DonohueiD, Will ForaniD, Ryland L. MilleriD, Eric FeczkoiD, Óscar Miranda-DomíngueziD, Alice M. GrahamiD, Eric EarliD, Anders J. PerroneiD, Michaela CordovaiD, Olivia J. DoyleiD, Lucille A. MooreiD, Greg ConaniD, Johnny Uriarte, Kathy SnideriD, Angela TamiD, Jianzhong Chen, Dillan J. NewboldiD, Annie ZhengiD, Nicole A. SeideriD, , Nico U.F. DosenbachiD

Abstract

Magnetic resonance imaging (MRI) continues to drive many important neuroscientific advances. However, progress in uncovering reproducible associations between individual differences in brain structure/function and behavioral phenotypes (e.g., cognition, mental health) may have been undermined by typical neuroimaging sample sizes (median N=25)1,2. Leveraging the Adolescent Brain Cognitive Development (ABCD) Study3 (N=11,878), we estimated the effect sizes and reproducibility of these brain-wide associations studies (BWAS) as a function of sample size. The very largest, replicable brain-wide associations for univariate and multivariate methods were r=0.14 and r=0.34, respectively. In smaller samples, typical for brain-wide association studies (BWAS), irreproducible, inflated effect sizes were ubiquitous, no matter the method (univariate, multivariate). Until sample sizes started to approach consortium-levels, BWAS were underpowered and statistical errors assured. Multiple factors contribute to replication failures4-6; here, we show that the pairing of small brain-behavioral phenotype effect sizes with sampling variability is a key element in wide-spread BWAS replication failure. Brain-behavioral phenotype associations stabilize and become more reproducible with sample sizes of N{gap}2,000. While investigator-initiated brain-behavior research continues to generate hypotheses and propel innovation, large consortia are needed to usher in a new era of reproducible human brain-wide association studies.

PDFDOI
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
182
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 1 (22 Aug 2020)

Brain-behaviour correlations in ABCD are tiny (median |r|=0.01), so samples of typical size (N=25) give inflated, irreproducible results; estimates only stabilise around N⪆2,000.

Brain-wide association studies (BWAS), which correlate individual differences in brain structure or function with behaviour, typically use a median of N=25 participants, and their findings often fail to replicate. Using the Adolescent Brain Cognitive Development (ABCD) Study with a Human Connectome Project (HCP) check, the paper measures effect sizes, sampling variability, error rates and out-of-sample replication as sample size grows from N=25 to N=3,928, for univariate and multivariate models.

Key results

  1. ShownBrain-behaviour associations are small: across cortical thickness and resting-state functional connectivity (RSFC) features at several resolutions and 41 phenotypes, the median correlation was |r|=0.01 and the strongest |r|=0.16.

    Full denoised ABCD sample (N=3,928): median |r|=0.01; top 1% |r|>0.06; strongest |r|=0.16 (RSFC edge with cognitive ability); largest univariate association that replicated out of sample |r|=0.14 (RSFC with NIH Toolbox crystallized intelligence). A less strictly motion-censored sample (N=9,753) gave the same maximum r=0.16 and 99th percentile |r|>0.06. · Brain-Wide Associations Cannot Be Estimated…; Univariate Brain-Behavioral Phenotype…

  2. ShownSampling variability alone makes small-sample estimates unreliable: at N=25 the same association can come out strongly positive or strongly negative and still be nominally significant, and noticeable spread remains at N=2,000.

    RSFC with cognitive ability (full-sample r=0.16): 99% CI r=-0.39 to r=0.62 at N=25, and 0.10-0.22 at N=2,000. Two N=25 subsamples in the default mode network example (cortical thickness with cognitive ability) gave r=0.64 vs r=-0.69, both p<0.001, against a full-sample r=0.08. Even across 100 split halves of N=1,964, the top 1% of correlations were inflated by r=0.043 on average. · Brain-Wide Associations Cannot Be Estimated…; Effect Sizes and Sampling Variability Replicate…; Empirical Quantification of Sampling…

  3. ShownTypical BWAS samples (N≤200) miss most true associations, inflate the ones they detect and often get the sign wrong; univariate models fitted on them never replicated out of sample, and 80% power for the top 1% of effects needs N=2,200 or more.

    N≤200 at p<0.05: false negative rate at least 76% (maximum power 24%) with a 6.1% false positive rate; 50% chance a significant correlation is inflated by at least 100%; at least 40% chance of a wrong sign; out-of-sample R2 −0.018 to 0.005. For 80% power at r=0.06: 2,200 subjects (p<0.05) or 9,500 (Bonferroni). At N=1,964 the top 0.5% explained 1% of variance (R2=0.01). · Statistical Errors Are Ubiquitous in Typically…

  4. ShownSmall effects and wide sampling variability are not artefacts of ABCD's children or its 21 sites: a single-site adult cohort gave a matching effect-size distribution at equal sample size, and one ABCD site varied like the rest.

    HCP (N=877 after denoising) max r=0.20 vs ABCD max r=0.16; ABCD subsampled to N=877 rose to 0.20 with a nearly identical effect-size distribution. Sampling variability was equivalent between HCP and ABCD, and between ABCD site 16 (N=603) and the remaining 20 sites (N=3,325). · Effect Sizes and Sampling Variability Replicate…; Human Connectome Project (HCP) Replication…; Single-site ABCD vs. Multi-site ABCD: Sampling…

  5. ShownMultivariate models replicated better than univariate ones but still needed large samples: support vector regression (SVR) from RSFC to cognitive ability gave the largest replicated effect with the full discovery set, while canonical correlation analysis (CCA) never replicated at N=200.

    SVR RSFC to cognitive ability r=0.34 (null r=0.10) at N=1,964 vs r=0.15 (null r=0.08) at N=200; cortical thickness SVR r=0.20 (null r=0.09) at N=1,814; CCA RSFC to NIH Toolbox r=0.22 (null r=0.09). RSFC CCA at N=200: all out-of-sample r<0.07 despite discovery r=0.35-0.66 (NIH Toolbox) and 0.32-0.66 (CBCL). Psychopathology did not replicate. Multivariate replicability tracked univariate effect size (r=0.79, p<0.001). · Multivariate Associations Can Improve…

Methods, data and reproducibility

ABCD participants were resampled with replacement at sizes from N=25 to 3,928 (1,000 draws per size for correlations, 100 bootstraps for multivariate models); cortical thickness and RSFC were correlated with 41 phenotypes, error, inflation and sign rates were scored against the full-sample result, and univariate fits, SVR and CCA were trained on matched discovery subsets and tested on the replication set.

  • ABCD Study RSFC and cortical thickness with 41 demographic, cognitive (NIH Toolbox) and mental-health (CBCL) measures; ARMS matched discovery and replication sets · human (children aged 9-10) · n = 3,928 RSFC (1,964 discovery + 1,964 replication); 3,604 cortical thickness · ABCD collection 3165; Annual Release 2.0 DOI 10.15154/1503209
  • Full ABCD behavioural and demographic sample (behaviour-behaviour correlations, ARMS split) · human (children aged 9-10) · n = 11,572 · ABCD 2.0 release
  • ABCD sample with less strict motion censoring (robustness check) · human (children aged 9-10) · n = 9,753
  • Human Connectome Project 1,200 Subject Data Release: RSFC with NIH Toolbox subscales (single-site replication) · human (adults aged 22-35) · n = 877 after denoising
Methods & tools
ABCD-BIDS pipelines (modified HCP pipelines), FreeSurfer, ANTs, FSL applywarp, Connectome Workbench, DCANBOLDproc, FIRMM, e1071 R package (SVR), Matlab 2019A (CCA, SVD), ABCDE (ABCD Boolean Capture Data Explorer), NIH Toolbox and CBCL
Reagents & environment
R 3.5.2 with the e1071 package for SVR; Matlab 2019A for CCA; ABCD-BIDS processing pipelines.
Experimental conditions
XSEDE Bridges system at the Pittsburgh Supercomputing Center; multivariate bootstraps were limited to 100 per size because of computational cost; no run times given.
Reproducibility gaps
Links to the study's analysis code and the ABCDE source were lost in extraction; ABCD data come from the NIMH Data Archive (collection 3165, Release 2.0) with ARMS subject lists promised there; the HCP data link is also missing.
Limitations (5)
  • Associations are plain correlations and canonical multivariate models without covariate adjustment, because the aim was to study sampling variability rather than systematic confounding. · stated by the authors
  • Multivariate analyses used only 100 bootstrap samples per size for computational reasons, and the analyses focus on mean estimates rather than full distributions. · stated by the authors
  • Measurable sampling variability remains even at N=2,000, and the best multivariate replication was still low-to-moderate for most phenotypes; psychopathology models did not replicate. · stated by the authors
  • Only cortical thickness and resting-state connectivity were tested, in children aged 9-10 plus an adult HCP check restricted to RSFC with NIH Toolbox scores, so other brain measures (for example task-evoked activity) are not covered. · noted in this summary
  • False negative, inflation and sign-error rates are scored against the full-sample estimate (N=3,928) as if it were the true value, although the paper itself shows estimates near that size still vary. · noted in this summary

Who should read:Read this if you design, review or fund MRI studies linking brain structure or connectivity to behaviour and want empirical sample-size, power and replication numbers from the ABCD cohort.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISLibedinsky '22Burns '23Spisák '22Murtha '25Kang '24Makowski '24
83% semantic matchNeuroscience
Ilan Libedinsky, Koen Helwegen, Udo Dannlowski+7 more·bioRxiv·Jul 5, 2022· 11 citations