How Well Does Structural MRI Classify Autism? A Leakage-Free, Site-Aware and Externally Validated Machine-Learning Evaluation in Two ABIDE Cohorts

Abstract

Background: Many studies report high accuracy for classifying autism spectrum disorder (ASD) from structural MRI, but small samples, single train-test splits, site effects and information leakage can inflate performance. We evaluated how well brain morphometry distinguishes autistic from typically developing (TD) individuals under rigorous, leakage-free validation. Methods: FreeSurfer-derived features (n = 280: regional thickness, area and volume; subcortical volumes; Yeo-network composites; asymmetry indices; global measures) from ABIDE II (804 participants; 341 ASD; 12 sites) were used to train logistic regression, linear support vector machine, random forest, XGBoost and multilayer perceptron classifiers. Confound regression (age, age2, sex, brain volume), site-wise centring and standardisation were learned within training folds only. Performance was assessed with nested 10-fold cross-validation, leave-one-site-out validation and external validation in ABIDE I (917 participants; 20 sites), and compared with full-pipeline permutation tests and a confound-only baseline. Results: Cross-validated discrimination was modest: the best model (regularised logistic regression on all features) reached an area under the ROC curve (AUC) of 0.60 [0.56, 0.64], significantly above chance (permutation p = .002) but far from clinically useful and below a confound-only baseline using age, sex, brain volume and site (AUC 0.63); non-linear models did not perform better (AUC 0.45-0.60 across models and feature sets). Pooled leave-one-site-out AUCs were 0.54-0.61, with large between-site variation, and AUCs in the external ABIDE I cohort were 0.52-0.58. Age-stratified models did not exceed AUC 0.60. Across 300 random 80/20 splits in young adults, test-set AUCs ranged from 0.21 to 0.90, showing that single-split estimates such as 0.74 can arise by chance. Conclusions: Under leakage-free, site-aware and externally validated evaluation, structural MRI morphometry classifies autism only slightly better than chance. Reports of high accuracy should be interpreted with caution, and the heterogeneity of autism argues for stratification and normative approaches rather than binary diagnostic classifiers.