henhaobio
Sign in

bioRxivNeurosciencePreprint

Reliable Evaluation of Transductive Population Graph Neural Networks for Multisite Autism fMRI

Keliang Wan, Zongqing CheniD, Guixian Liu, Biantian Yu, Qian Zhang, Fanghong ZhangiD, Ning ZhongiD, Hongzhi KuaiiD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Reliable population-graph evaluation requires separating cohort-conditioned discrimination from performance at an unseen site. We evaluate a recovered site-aware graph neural network in ABIDE-I (871 participants, 20 sites), distinguishing archived intervention results from newly recomputed controls. Archived cohort AUC was approximately 0.94, with high discrimination retained by site-only connectivity. A reconstructed ten-fold empirical site-prior classifier achieved pooled out-of-fold AUC 0.506 (95\% site-resampling interval [0.418, 0.546]); simple site proportions therefore did not reproduce the high cohort result. In a budget-matched support-location experiment, the across-site mean AUC difference was +0.252 [0.184, 0.322] with directed edges but -0.003 [-0.020, 0.015] after symmetrisation, demonstrating implementation sensitivity rather than a stable support-location advantage. A supplementary leave-one-site-out analysis compared scoring-time population-edge access after the same source-site training procedure. Mean site-macro AUC was 0.577 with access and 0.580 without it; the paired difference was -0.0037 [-0.0179, 0.0111]. These observations do not establish equivalence or a complete mechanism. Explicit access contracts, compatible estimators and run-level provenance are necessary to interpret high transductive cohort performance.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 5 (18 Oct 2026) · CC BY 4.0

Site-aware population GNN on ABIDE-I: cohort AUC about 0.94 (kept with site-only edges) was not reproduced by a site-prior classifier (0.506); a directed-edge support gain (+0.252) vanished when symmetrised (−0.003).

Transductive population graphs let test subjects' imaging and phenotypes shape graph construction and message passing, so cohort cross-validation may report discrimination unavailable at a new site; a recovered model reached ABIDE-I cohort AUCs of approximately 0.90 to 0.96, above typical controlled multisite values. Instead of a new architecture, the work intervenes on that model's graph sources, features, supervision and heads to see which pathways sustain the score.

Key results

  1. ShownTransductive cohort discrimination was high and stayed high when the population graph used only site-defined edges, but dropped sharply with sex-only, random or no population edges; the pattern held under both C-PAC and NIAK preprocessing.

    Archived pooled out-of-fold (OOF) AUC with site-clustered interval: full model 0.941 (0.867 to 0.976), site-only 0.948 (0.875 to 0.982); sex-only 0.565, random 0.587, subject-only 0.587. NIAK: full 0.946, site-only 0.958, subject-only 0.570. Separately recomputed control runs: full 0.949842, site-only 0.955977. Re-implementation matched recovered outputs to within 7 × 10^−7. · 4.1. Implementation fidelity and cohort…; 4.2. Graph sources and demographic controls; Tables

  2. SuggestedCutting train-to-test edges, rewiring the graph, removing feature content, masking same-site supervision, or swapping the dual-channel sex-relation head each lost most of the high AUC, whereas within-site shuffling of fused features did not.

    Archived runs: train-test edge removal 0.562 (0.498 to 0.606), degree-preserving rewiring 0.597, constant features 0.600, within-site fused-feature shuffle 0.960; site-wise supervision masking 0.480 (871 subjects). Heads: dual-channel 0.949 vs single-channel TransformerConv 0.554, GAT 0.516, GCN 0.499, no-graph MLP 0.579; identical edges in both branches 0.651; Parisot-GCN-C comparator 0.656. · 4.3. Feature and train–test pathway…; 4.5. Architecture-specific dependence; Graph and feature controls; Architecture and supervision controls

  3. ShownSimple site and demographic explanations did not reproduce the cohort AUC: a training-only site-prevalence classifier and site/sex logistic regression both scored near 0.5.

    Empirical site-prior (ASD fraction among a site's optimization-training subjects), all 871 subjects scored: pooled OOF AUC 0.5056 (site-resampling 95% interval 0.4176 to 0.5464). Regularized site+sex logistic regression 0.5131 (0.425 to 0.555), with interaction 0.5006; site as node feature 0.504; site-by-diagnosis Cramér's V ≈ 0.16 (chi-squared p = 0.225). · 4.2. Graph sources and demographic controls; Empirical site-prior baseline

  4. ShownA budget-matched experiment comparing same-site with other-site labelled support gave a large positive effect only with directed population edges; after symmetrising the edges the effect disappeared, so it reflects implementation rather than a stable same-site advantage.

    Across-site mean ΔAUC (ON minus OFF; 18 sites, 88 site-fold cells, three seeds): directed edges +0.252 (site-resample 95% interval +0.184 to +0.322), positive at 17/18 sites, sign test p = 1.5 × 10^−4; symmetrised −0.003 (−0.020 to +0.015), positive at 8/18; signs agreed at 7/18 sites. Directed per-seed means +0.217, +0.256, +0.283. · 4.6. Paired support-location and implementation…; 3.4. Paired support-location and sensitivity…; Paired support-location endpoint

  5. SuggestedWith whole sites held out, discrimination was modest, and attaching target subjects through cross-site population edges at scoring time made no clear average difference.

    Leave-one-site-out, 20 sites, three fixed seeds: mean site-macro AUC 0.5768 with scoring-time edge access vs 0.5805 without; paired difference −0.0037 (95% site-resampling interval −0.0179 to +0.0111); 29/60 site-seed differences positive. The authors state this is not an equivalence result. · 4.4. Supplementary unseen-site access comparison; 3.1. Cohort and information-access regimes; Cohort and site-held-out endpoints

Methods, data and reproducibility

A recovered site-aware population graph neural network (SA-PopGNN: GCN/SAGPool subject encoder plus a two-channel TransformerConv head over same-sex and different-sex edges) was re-implemented to match its outputs, then probed on ABIDE-I by transductive ten-fold cross-validation with graph, feature, supervision and head interventions, a budget-matched paired support experiment and leave-one-site-out scoring.

  • ABIDE-I canonical cohort, resting-state fMRI functional connectivity (C-PAC and NIAK, filt_noglobal; AAL-116, CC200 and HO-111 atlases) · human (autism (ASD) and typically developing participants) · n = 871 participants (403 ASD, 468 typically developing) from 20 sites
  • Paired support-location subset (eligible site-fold cells of ABIDE-I) · human · n = 836 subjects, 18 sites, 88 site-fold cells
  • Released code, frozen results and selected inputs for recomputation (Zenodo v1.2.1) · 10.5281/zenodo.23445229
Methods & tools
SA-PopGNN (re-implementation), C-PAC, NIAK, AAL-116 / CC200 / HO-111 atlases, TransformerConv, SAGPool / DiffPool-style pooling, ChebConv, Parisot-GCN-C, networkx double_edge_swap, StandardScaler / PCA, random forest and linear SVM, integrated gradients
Software & environment
Archive version 1.2.1 corresponds to GitHub tag v1.2.1 (commit 84933c3dfc2bdb4b653b3eaacc938cdcba04f540); the Python/Matplotlib figure script is supplied with its dependency version.
Assay conditions & compute
Training ran on GPU (authors note small nondeterminism, so reported values refer to frozen artifacts); hardware, runtime and memory not stated.
Reproducibility gaps
Participant-level predictions and reconstructed masks are unreleased, limiting individual-level recomputation; original inner-validation masks and the NIAK cache builder are unavailable; ABIDE-I imaging is not redistributed; the label-permutation re-run is not fully reconstructible.
Limitations (6)
  • Only ABIDE-I and one site-aware model family were studied; seed repetitions were limited and not every intervention was run with three seeds; small held-out sites yield uncertain site-level estimates. · stated by the authors
  • The support-location endpoint was defined after training (not pre-registered), relies on one support draw per cell and on edge construction, and cannot separate supervision location from subject identity and covariate composition. · stated by the authors
  • Bootstrap intervals condition on saved predictions and fixed seeds, omitting retraining variability; non-significant contrasts are not equivalence tests, and historical p-values and Holm adjustments are kept only for traceability. · stated by the authors
  • Run provenance is incomplete: archived headline runs cannot be tied to recomputed OOF files (0.9405 vs 0.939609 vs 0.949842), the mechanism-control cache lacks a campaign-time hash, and integrated-gradients attribution was unstable (top-20 ROI overlap 0.23). · stated by the authors
  • The leave-one-site-out comparison varies only scoring-time edge access over three seeds; training-time target access, prospective single-subject prediction and cohort-composition shifts were not tested. · stated by the authors
  • No imaging-only or simpler graph baseline was rerun under the new leave-one-site-out protocol (historical figures are explicitly set aside), so the 0.577 site-macro AUC lacks a same-protocol reference point. · noted in this summary

Read this if you build or review population graph models for multisite neuroimaging and need controls that separate transductive cohort performance from performance at an unseen acquisition site.

Citations
0

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISYousedian '22Li '21Kunda '20Klepl '22Abraham '16Murugesan '20
59% semantic matchNeuroscience
Ali Yousedian, Farzaneh Shayegh, Zeinab Maleki·bioRxiv·Jun 23, 2022· 2 citations