Neural prediction decorrelation reveals that adversarial robustness substantially improves DNN prediction accuracy across the entire human auditory cortex
Abstract
Sensory neuroscientists seek to model the neural computations that encode complex stimuli. Distinct encoding models often make similar predictions for natural stimuli such as speech, posing a challenge for model comparison. We developed a method to synthesize stimuli that decorrelate model predictions across a neural population, termed neural prediction decorrelation (NPD). Using fMRI responses to NPD sounds, we compared standard and adversarially robust deep neural network models of human auditory cortex. Prediction accuracy for NPD sounds was substantially better for the adversarially robust model in every region tested, an effect completely masked with natural sounds. Population responses to natural and synthesized NPD sounds shared an interpretable low-dimensional organization that was reproduced by the robust encoding model. NPD provides a general approach for comparing encoding models and reveals that adversarial robustness expands the predictive power of DNNs beyond natural stimuli, which is likely critical for targeting population activity through stimulus synthesis.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 2
- Organism
- Assay
- Research Focus
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (11 Oct 2026) · CC BY 4.0
Sounds synthesized to decorrelate model predictions across human auditory cortex show an adversarially robust CochResNet50 predicts fMRI responses far better than its standard counterpart, a gap hidden with natural sounds.
Competing encoding models, including standard and adversarially robust DNNs, often predict responses to natural stimuli almost equally well, so neural data cannot say which is better. NPD optimizes stimuli to decorrelate competing models' predictions for every channel of a neural population at once, whereas earlier behavioral methods only split model decisions and earlier neural synthesis work did not target model comparison.
Key results
ShownWith encoding models fit only on natural sounds, the robust model kept strong accuracy on NPD sounds while the standard model fell to near zero, so the robust model was better in every auditory cortex region and every participant.
S2026 fMRI study (9 participants, 60 NPD + 120 natural sounds); models fit on 60 natural, tested on 60 NPD and 60 held-out natural sounds. Robust vs standard dissimilarity on NPD sounds (0 identical, 1 maximal): medial Heschl's gyrus 0.539 (0.335, 0.822), lateral 0.680 (0.611, 0.744), planum polare 0.976, planum temporale 0.916, superior temporal gyrus 1.000 (0.904, 1.000). · NPD sounds reveal that adversarial robustness…; S2026 experiment: fMRI responses to NPD and…; Figure legends
ShownOn natural sounds the two networks were nearly indistinguishable. Their prediction accuracy was very similar, and their predictions correlated with each other about as strongly as two fits of the same network to noisy data.
165 natural sounds, 30 participants (NH2015, B2021), 55 test sounds: accuracy dissimilarity 0.007 to 0.031 across five regions, e.g. medial Heschl's gyrus 0.023 (0.013, 0.033); Across-DNN vs Same-DNN prediction-correlation dissimilarity 0.001 to 0.022. Replicated on 60 held-out natural sounds in the S2026 cohort. · Encoding models derived from standard and robust…; NPD sounds decorrelate model predictions across…; NPD sounds reveal that adversarial robustness…; DNN encoding models
ShownSounds synthesized from one cohort's encoding models decorrelated robust and standard predictions in a separate cohort that was never used for synthesis.
60 NPD sounds from NH2015 models (10 participants), evaluated with models fit to B2021 (20 participants): group median correlation of robust and standard predictions across voxels 0.102 for NPD vs 0.797 for natural sounds (about eightfold lower); every reliably sound-responsive voxel showed a lower correlation for NPD sounds. · NPD sounds decorrelate model predictions across…; Neural prediction decorrelation
SuggestedNPD sounds evoked cortical response patterns that overlapped those of natural sounds and shared a low-dimensional organization by category and acoustics. Within that space the two models diverged for NPD sounds, and the robust model's predictions lay closer to measured responses.
Response-pattern correlations between NPD and natural sounds were as high as within either set (9 participants). In UMAP of the top 6 principal components, speech-like NPD #43 clustered near natural speech and NPD #33 near 'Running water'. Distance comparisons in the 6-component space (medians, 5,000-resample bootstrap CIs) are shown in figures, with no numbers given in the text. · Overlapping cortical responses and divergent…; Measuring similarity of voxel response patterns
ClaimedThe authors argue that adversarial training widens the range of stimuli DNNs predict well enough to control neural populations in human auditory cortex without regularizing constraints. They say this has not, to their knowledge, been shown in auditory cortex.
This is the authors' interpretation in the Discussion, drawn from the NPD and shared-subspace results. They contrast it with a cited study in which a standard DNN synthesizing sounds for speech- and music-selective regions needed smoothness constraints and achieved only partial activation relative to natural sounds. · Benefits of adversarial robustness for neural…; Leveraging stimulus synthesis to test…
Methods, data and reproducibility
Gradient descent on 60 Gaussian-noise-initialized 2-second sounds pushed, for every auditory cortex voxel, the covariance between standard and robust CochResNet50 encoding-model predictions toward zero and their variances toward 5 times natural-sound levels. The encoding models were voxelwise ridge regressions on activations from the best-predicting DNN layer, and the sounds were then tested in new fMRI participants.
- NH2015 fMRI responses to 165 natural sounds (2 s each, 11 categories); encoding models used to synthesize NPD sounds · human · n = 10 participants
- B2021 fMRI responses to the same 165 natural sounds; held-out cohort for testing decorrelation · human · n = 20 participants
- S2026 new fMRI experiment: 60 NPD sounds plus 120 natural sounds (a subset of the 165) · human · n = 9 participants
- Methods & tools
- CochResNet50 (standard and adversarially robust), ridge regression, fMRIPrep 25.1.4, FSL, FreeSurfer, PyTorch, Adam optimizer, UMAP, PCA, 3T Siemens Trio and Prisma MRI scanners, Sensimetrics S14 earphones
- Data & accessions
- https://doi.org/10.5281/zenodo.22038940https://labsites.rochester.edu/neural-prediction-decorrelation
- Software & environment
- fMRIPrep 25.1.4 with default parameters (S2026); FSL (NH2015, B2021) and FreeSurfer (all experiments); synthesis in PyTorch, version not stated.
- Reproducibility gaps
- The preprocessing code and the original (defaced) and preprocessed fMRI data are described as being released upon publication. Scan parameters for NH2015 and B2021 are given only in the earlier publications.
Limitations (5)
- Responses are time-averaged per 2-second sound (as fMRI temporal resolution is coarse), so the comparison cannot address whether the models differ in predicting response dynamics. · noted in this summary
- The loss relies on the covariance of model predictions. Alternative dependency measures such as mutual information could be used, but are hard to estimate from small stimulus sets. · stated by the authors
- Decorrelation is optimized per voxel across sounds. Explicitly decorrelating population response patterns, and biasing synthesis toward naturalistic sounds, are left for future work. · stated by the authors
- The measured gain in response variance for NPD sounds was smaller than predicted, plausibly because measured responses include stimulus-independent noise. · stated by the authors
- Section S6 says the distance between the two models' predictions was smaller for NPD than for natural sounds. This contradicts the Fig. 4D legend and the surrounding argument that NPD predictions diverge. It is probably a wording error, but Fig. 4D should be checked. · noted in this summary
Who should read:Read this if you compare encoding models of sensory cortex on natural stimuli and need stimuli that separate them, or if you study adversarially robust audio DNNs as models of auditory cortex.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions
Greta Tuckute, Jenelle Feather, … Josh H. McDermott·PLoS BiologyPeer-reviewed·Dec 13, 2023
- How well do models of visual cortex generalize to out of distribution samples?
Yifei Ren, Pouya Bashivan·PLoS Computational BiologyPeer-reviewed·May 31, 2024
- Can Deep Learning Provide a Generalizable Model for Dynamic Sound Encoding in Auditory Cortex?
Jacob R. Pennington, Stephen V. David·bioRxiv·Jun 10, 2022
- How predictive can be predictions in the neurocognitive processing of auditory and audiovisual speech? A deep learning study.
Thomas Hueber, E. Tatulli, … Jean‐Luc Schwartz·bioRxiv·Nov 18, 2018
- Neural signals to violations of abstract rules using speech-like stimuli
Yamil Vidal, Perrine Brusini, … Tristan Andres Bekinschtein·eNeuroPeer-reviewed·Sep 1, 2019
- Cortical response to naturalistic stimuli is largely predictable with deep neural networks
Meenakshi Khosla, Gia H. Ngo, … Mert Rory Sabuncu·Science AdvancesPeer-reviewed·May 28, 2021