henhaobio
Sign in

bioRxivBioinformaticsPreprint

Controlling for effects of confounding variables on machine learning predictions

Richard DingaiD, Lianne SchmaaliD, Brenda W.J.H. PenninxiD, Dick J. VeltmaniD, André F. MarquandiD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

Machine learning predictive models are being used in neuroimaging to predict information about the task or stimuli or to identify potentially clinically useful biomarkers. However, the predictions can be driven by confounding variables unrelated to the signal of interest, such as scanner effect or head motion, limiting the clinical usefulness and interpretation of machine learning models. The most common method to control for confounding effects is regressing out the confounding variables separately from each input variable before machine learning modeling. However, we show that this method is insufficient because machine learning models can learn information from the data that cannot be regressed out. Instead of regressing out confounding effects from each input variable, we propose controlling for confounds post-hoc on the level of machine learning predictions. This allows partitioning of the predictive performance into the performance that can be explained by confounds and performance independent of confounds. This approach is flexible and allows for parametric and non-parametric confound adjustment. We show in real and simulated data that this method correctly controls for confounding effects even when traditional input variable adjustment produces false-positive findings.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (18 Aug 2020) · CC BY 4.0

Regressing confounds from inputs leaves confound information machine learning models exploit; adjusting the predictions post hoc instead held false-positive rates near 5%, versus up to 100% with input adjustment.

Neuroimaging machine learning (ML) predictions can be driven by confounds such as scan site or age, and the usual fix (least-squares regression of confounds out of each input) misses nonlinear, variance, shape, multivariate and outlier-driven effects that learners exploit. The paper adapts pre-validation from microarray studies to confound control, testing predictions against confounds in a post-hoc regression and splitting explained variance or deviance into confound-only, prediction-only and shared parts.

Key results

  1. ShownIn simulated regression and classification data where confounds explain all outcome signal, regressing confounds from the inputs (using parameters estimated on the training set, following Snoek et al. 2019) did not prevent significant above-chance prediction.

    False-positive rate at p < 0.05 could reach 100%, depending on confounding strength (regression weights 0.1, 0.2, 0.3; classification P(y=1) of 0.6, 0.7, 0.8). Each dataset had 200 samples. Models were RBF support vector regression (C=1) and Gaussian process classification, tested on a held-out half (parametric) and in 5-fold cross-validation (permutation). · EXPERIMENTS; RESULTS

  2. ShownIn real ABIDE ROI thickness data (cortical, white-matter and non-cortical ROIs) with an artificial outcome generated purely from age, adjusting the inputs for age still produced frequent false positives.

    False-positive rate as high as 72% after age adjustment of inputs; 97 ROIs, 530 healthy controls aged 6 to 56 years (median 14.66, IQR 8.2); age-confounding weights of 3, 4 and 5. · EXPERIMENTS; RESULTS

  3. ShownAdditionally adjusting the ML predictions (entering them with the confounds into a linear regression for continuous outcomes or a logistic regression for binary ones, on top of the input adjustment) kept false-positive rates near the nominal 5% in every tested scenario.

    False-positive rate around 5% (nominal p < 0.05) at low, medium and high confounding in simulated regression, simulated classification and ABIDE. This held both with a parametric test on a held-out half and with a permutation test in 5-fold cross-validation. Per-scenario rates appear only in Figure 6. · EXPERIMENTS; RESULTS; DISCUSSION

  4. ShownIn UK Biobank, ridge regression on regional brain volumes captured a small part of the variance in fluid intelligence beyond brain size and, in a separate model, beyond education length.

    Test-set predictions: R^2=0.035 (p < 0.001), of which 0.014 was shared with brain size and 0.021 was unique (p < 0.001); brain size added 0 (p=0.290). Against education length (cubic spline, 5 knots): R^2 0.039 for predictions vs 0.079 for education, with the partial effect of predictions still significant. Training n=9,630, test n=5,167. · Dataset; Method; Results

  5. ClaimedNonlinear learners and support vector machines are argued to exploit confounding left after input adjustment (even ComBat-style location-scale correction), so some published results may be confound-driven; penalized linear and logistic regression should be unaffected, provided the inputs are sufficiently adjusted.

    This rests on schematic examples (Figures 1 and 2; the hinge loss of support vector machines is less sensitive to outliers than squared error) and on the kernel-model simulations. No published study is re-analysed. The experiments used only training-set least-squares input adjustment, so ComBat and penalized linear models were not tested. · PROBLEMS OF INPUT ADJUSTMENT; DISCUSSION

Methods, data and reproducibility

Any ML model's test-set predictions enter a linear (continuous outcome) or logistic (binary) regression alongside confounds; added value is tested by partial correlation/F test or likelihood ratio, and R^2 or deviance explained (D^2) is split into confound-only, prediction-only and shared parts. In cross-validation, a permutation test shuffles input rows while keeping outcome-confound pairs intact.

  • UK Biobank 2018 release: regional brain volumes from structural MRI (FSL FAST) with fluid intelligence scores; brain size and age at completing full-time education as confounds · human (UK Biobank participants) · n = 21,407 before exclusions; training set 9,630, test set 5,167
  • ABIDE ROI thickness data (97 ROIs from ANTs Desikan-Killiany and FreeSurfer) via the Preprocessed Connectome Project, with a synthetic binary outcome generated from age · human (ABIDE healthy control subjects) · n = 530 healthy controls aged 6 to 56 years
  • Simulated regression data: uniform confound (0 to 100), one input with heteroscedastic noise, outcome driven by the confound only; three confounding levels · n = 200 per simulated dataset
  • Simulated classification data: binary confound setting the correlation (0.9 or -0.9) of two bivariate-normal inputs, outcome driven by the confound only; three confounding levels · n = 200 per simulated dataset
Methods & tools
glmnet (ridge regression), kernlab (support vector regression, Gaussian process classification), FSL FAST, funpack v2.0.1, ANTs, FreeSurfer
Software & environment
funpack v2.0.1 is the only versioned tool; no versions are given for glmnet or kernlab, and no container or requirements file is described.
Reproducibility gaps
No data links or accessions are given for UK Biobank or ABIDE; the number of simulation repetitions per scenario is not stated.
Limitations (5)
  • Both the regression and classification versions ignore miscalibration, such as over- or underconfident probabilities; they ask whether predictions carry information beyond the confounds, not whether their absolute error beats a confounds-only model. · stated by the authors
  • Parametric p-values are valid only with an independent test set. Under cross-validation, fold errors are dependent, so a permutation test is required. It must shuffle input rows rather than outcomes, because shuffling outcomes breaks the confound-outcome link. · stated by the authors
  • Which variables count as confounds depends on the study goal (e.g., current symptom severity when predicting remission). Nonlinear confound effects must be modelled with splines or generalized additive models; binning continuous confounds leaves residual confounding. · stated by the authors
  • Simulations use 200 samples per dataset and one learner per task (RBF support vector regression, Gaussian process classification). The number of simulation repeats and the exact per-scenario false-positive rates are not given in the text. · noted in this summary
  • Real-data validation rests on a single cohort (ABIDE healthy controls) with a synthetic age-driven outcome. The claim that penalized linear models are unaffected by residual confounding is not tested. · noted in this summary

Read this if you train predictive models on neuroimaging or other high-dimensional data with confounds such as site, age or motion and need a valid test of what the predictions add beyond them.

Citations
76

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISWang '16Wang '18Rogozhnikov '22Vliet '19Rosenblatt '24Raab '26
65% semantic matchBioinformatics
Haohan Wang, Jingkang Yang·IEEE International Conference on Bioinformatics and Biomedicine (BIBM)·Dec 1, 2016· 20 citations