henhaobio
Sign in

bioRxivNeurosciencePreprint

What is the test-retest reliability of common task-fMRI measures? Newempirical evidence and a meta-analysis

Maxwell L. ElliottiD, Annchen R. KnodtiD, David R. IrelandiD, Meriwether L. Morris, Richie PoultoniD, Sandhya RamrakhaiD, Maria L. Sison, Terrie E. MoffittiD, Avshalom CaspiiD, Ahmad R. HaririiD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task-fMRI is a major focus of biomarker development; however, the reliability of task-fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments (N=1,008) revealed poor overall reliability (mean ICC=.397). Second, the test-retest reliabilities of activity in a priori regions of interest across 11 common fMRI tasks collected in the context of the Human Connectome Project (N=45) and the Dunedin Study (N=20) were poor (ICCs=.067-.485). Collectively, these findings demonstrate that common task-fMRI measures are not currently suitable for brain biomarker discovery or individual differences research. We review how this state of affairs came to be and highlight avenues for improving task-fMRI reliability.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 3 (16 Jan 2020) · CC BY 4.0

Task-fMRI regional activation has poor test-retest reliability: meta-analytic mean intraclass correlation .397 across 77 un-thresholded substudies, and .067 to .485 for 11 tasks in HCP and Dunedin retest samples.

Task-fMRI is a major route to brain biomarkers of disease risk, but an unreliable measure cannot predict outcomes, and earlier single-study estimates came from small samples with varied methods and sometimes conflicting conclusions. This work combines a systematic meta-analysis with pre-registered retest analyses of 11 tasks in two recent datasets, benchmarked against structural MRI.

Key results

  1. ShownAcross the published literature, the average test-retest intraclass correlation coefficient (ICC) for task-fMRI regional activation is in the poor range. At that level, brain-behavior correlation studies would need hundreds of participants.

    77 substudies reporting un-thresholded ICCs (median N = 17): random-effects multilevel mean ICC .397 (95% CI .330-.460; P < .001); 56% of values poor (<.4), 24% fair, 20% good or excellent. Power curves show that at ICC .397, 80% power to detect r = .3 needs N > 214, and r = .1 needs N > 2,000. · Reliability of Individual Differences in…; 3) More data from more subjects; Statistical Analyses

  2. ShownStudies that reported only ICCs passing a significance or magnitude threshold give inflated reliability. The only other significant moderator was ROI location, with cortical regions above subcortical ones. Task type, design, length, retest interval, ROI type and sample type did not moderate reliability.

    13 thresholded substudies: meta-analytic ICC .705 (95% CI .628-.768), 1.78 times the un-thresholded estimate. Moderator model on all 1,146 ICCs: thresholding QM = 6.531, P = .010, β = .140; cortical vs subcortical QM = 114.476, P < .001, β = .259. Egger test found no publication bias (Z = .707, P = .480). · Reliability of Individual Differences in…; Statistical Analyses

  3. ShownPre-registered analyses of two recent retest samples scanned on modern 3T scanners found poor reliability for activation in each task's a priori target region. No task reached the good range.

    Human Connectome Project (HCP; N=45, 7 tasks): target ROI mean ICC .251 (95% CI .142-.360); only the language task exceeded poor (ICC .485). Dunedin Study (N=20, 4 tasks): mean ICC .309 (95% CI .145-.472). Functional 5mm-sphere ROIs: .381 (HCP) and .325 (Dunedin). Hemisphere-specific ROIs and an unrelated HCP subsample (N=26, mean .301) gave similar values. · Reliability of Individual Differences in…; Reliability of regional activation in the Human…; Reliability of regional activation in the…

  4. SuggestedRegions a task is designed to engage were no more reliable than other regions, even though every task produced the expected robust group-level activation in its target.

    Non-target ROI mean ICC .239 in HCP (vs target .251, P = .474) and .193 in Dunedin (vs .309, P = .140). Group activation maps confirmed target activation for all 11 tasks, while voxel-wise ICC maps for the three tasks shared by both studies showed poor between-subject reliability across the whole brain, not only in target ROIs. · Group-level activation; Reliability of regional activation in the Human…; Reliability of regional activation in the…; Figure legends

  5. ShownStructural MRI measures in the same participants were highly reliable, which the authors take to mean the problem is the task paradigms rather than MRI itself.

    Average cortical thickness ICC .953 (HCP) and .939 (Dunedin); total surface area .999 and .996; subcortical volumes averaged .903 (range .791-.984) and .931 (range .767-.979). Parcel-wise cortical thickness across 360 parcels averaged .886 and .846. · Reliability of structural measures; The two disciplines of fMRI research

Methods, data and reproducibility

A PRISMA-guided systematic review and random-effects multilevel meta-analysis (metafor in R) of published task-fMRI test-retest ICCs, with moderator tests. It is paired with pre-registered ICC(3,1) analyses of activation in a priori anatomical and functional regions of interest for 11 tasks in HCP and Dunedin retest scans, benchmarked against cortical thickness, surface area and subcortical volume.

  • Published task-fMRI test-retest studies from a Google Scholar search (to April 1, 2019): 56 articles, 1,146 ICCs, 90 substudies, 66 paradigms · human (healthy and clinical participants) · n = 1,088 unique participants (abstract states N=1,008)
  • HCP test-retest subsample: n-back, gambling, motor, language, social, relational and face-matching tasks; custom 3T Siemens scanner; mean interval about 140 days · human (Human Connectome Project; participants aged 25-35) · n = 45 retest participants of 1,206; 26 unrelated in a sensitivity analysis
  • Dunedin Study retest subsample at age 45: face-matching, Stroop, monetary incentive delay and face-name encoding tasks; 3T Siemens Skyra; mean interval 79 days · human (New Zealand birth cohort) · n = 20 retest participants of a 1,037-member cohort
Methods & tools
metafor (R package), AFNI 3dICC_REML.R, pwr.r.test (R), ICC(3,1) (Shrout and Fleiss), Egger random-effects regression test, threshold-free cluster enhancement, Google Scholar, 3T Siemens Skyra
Reproducibility gaps
The analysis plan was pre-registered on a Google Sites project page. Preprocessing and software details are left to the Supplement, which is not part of this text, and access terms for the raw Dunedin imaging data are not stated.
Limitations (5)
  • The retest samples are small (N=45 and N=20). The authors caution against strong conclusions about any specific task and note that correlations stabilise only at around 150 observations. · stated by the authors
  • The meta-analysis pools many tasks, samples, scanners and pipelines (some outdated), so a single pooled estimate could mask paradigms that are truly reliable. · stated by the authors
  • Test-retest ICC is a conservative estimate that habituation and rapid state fluctuations can lower. Internal consistency is offered as a cheaper complement but is not sufficient for prognostic biomarkers. · stated by the authors
  • Most source studies reported no standard error for their ICCs, so standard errors were approximated with a Fisher r-to-Z transform, and some ICCs were judged from bar graphs. · noted in this summary
  • Reliability was tested for univariate contrast (change-score) activation, mainly ROI-averaged with supplementary voxel-wise ICC maps. The multivariate, latent-variable and connectivity measures the authors recommend as remedies were not evaluated on these data. · noted in this summary

Read this if you use task-fMRI activation in regions of interest as an individual-differences measure or biomarker and need reliability estimates and their sample-size consequences before designing or interpreting a study.

Citations
67

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISFröhner '19Holiga '18Nikolaidis '22Taxali '20Kennedy '22Gell '23
79% semantic matchNeuroscience
Juliane Hilde Fröhner, Vanessa Teckentrup, Michael N. Smolka+1 more·NeuroImage·Mar 28, 2019· 98 citations