From sound to source: Human and model recognition of environmental sounds
Abstract
Our ability to recognize sound sources in the world is critical to daily life, but is not well documented or understood in computational terms. We developed a large-scale behavioral benchmark of human environmental sound recognition, built models of sound recognition that operate on the acoustic signal, and used the benchmark to compare models to humans. The behavioral benchmark measured how sound recognition varied across source categories, audio distortions, and concurrent sound sources, all of which influenced recognition performance in humans. Artificial neural network models trained to recognize sounds in multi-source scenes reached near-human accuracy and qualitatively matched human patterns of performance in many conditions. By contrast, classifiers trained to recognize sounds from traditional models of the cochlea and auditory cortex produced worse matches to human performance. Models trained on larger datasets exhibited stronger alignment with both human behavior and brain responses. The results suggest that many aspects of human sound recognition emerge in systems optimized for the problem of real-world recognition.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 3
- Organism
- Assay
- Research Focus
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (1 Aug 2026) · CC BY 4.0
Neural networks trained to detect sound categories in synthetic multi-source scenes approach human accuracy on the EnvAudioEval behavioral benchmark and track human patterns better than cochlear or cortical filterbank baselines.
Human recognition of everyday sounds has been studied with small stimulus sets and few candidate models, and machine-hearing networks had not been systematically compared with human behavior. EnvAudioEval spans 2176 sounds and 68 distortions (earlier studies used 70 or 168 sounds and, in one case, 20 distortions), and the authors present it as the first large-scale human data on multi-source scenes and category confusions.
Key results
ShownHuman detection of sound categories fell as more sources were added but stayed above chance with five. It varied reliably across the 51 categories and across distortions. Removing frequency bands hurt most; temporal manipulations such as local time reversal or dilation hurt little.
Split-half reliability across categories ρ = 0.97 and across distortion conditions ρ = 0.95 (p < 0.001). Single-channel noise vocoding still gave above-chance d', and a 3 Hz temporal modulation lowpass left d' of about 1. Cueing the category before the scene helped overall (F(1,107)=17.09, p<0.001) but not for single sources (t(107)=0.52, p=0.60). · Experiment 1a: Human recognition declines with…; Experiment 1a: Humans show reliable variation in…; Experiment 1c: Cueing manipulation; Humans show reliable variation in recognition…
ShownNetworks trained on the synthetic scenes came much closer to human accuracy across scene sizes and categories than the linear filterbank baselines did. The three pretrained models fit best, and SSASTPretrained was not significantly below humans across categories.
All models showed the scene-size decline (F(4, 250) > 13.22). RMSE to human d' across scene sizes: Coch 1.33, ST 0.91, CochCNN 0.44, CochCNNPretrained 0.42, VGGishPretrained 0.39, SSASTPretrained 0.38. Category-wise human-model correlation .69 to .88 for the networks, under the human ceiling (ρ = 0.99). SSASTPretrained vs humans t(50) = -0.91, p = 0.37; every other model fell below humans (p<0.04). · Human and model recognition degrades similarly…; Pretrained models best replicate human…
ShownModels tended to capture the qualitative effect of each distortion, but all were less robust than humans. The gap was largest for high-, low- and band-pass audio filtering; humans and models were about equally robust to modulation filtering.
Paired t-tests over distortion type and level: every model below humans (p<0.001). Human-model correlation across conditions r = 0.59 to r = 0.93, highest for the pretrained models. A spectrotemporal stage improved robustness (CochSTVGGish over CochVGGish, t(67)=6.25, p<0.001); SSASTPretrained vs VGGishPretrained was not significant (t(67)=1.09, p=0.28). · Pretrained models best reproduce human…
ShownModels only partly reproduced human category confusions and exemplar-level hit rates. Every model correlated positively with humans and every network beat both baselines, but all stayed below the human noise ceiling.
Off-diagonal confusion correlation: Coch and ST ρ = 0.19; CochCNN, CochVGGish and CochSTVGGish ρ = 0.31; CochCNNPretrained ρ = 0.29; VGGishPretrained ρ = 0.36; SSASTPretrained ρ = 0.33. Exemplar hit-rate correlation: Coch ρ = 0.07, ST ρ = 0.28, CochCNN ρ = 0.39, VGGishPretrained ρ = 0.48, SSASTPretrained ρ = 0.43. · Task-optimized models replicate human patterns…; Task-optimized models replicate human…
SuggestedOn auditory-cortex fMRI responses to 165 natural sounds, all trained networks predicted voxels better than the baselines. The pretrained models did best on both regression and RSA, and models with better behavioral alignment tended to show better brain alignment.
Voxelwise ridge regression and representational similarity analysis on 7,694 voxels from 8 participants of a previously published dataset, with the best stage chosen on held-out splits. In RSA only one network beat the ST baseline. The behavior-brain link is described as a tendency across models with no statistic reported, and alignment was far from perfect. · Better models of behavior show higher brain…; General; Voxelwise modeling: Aggregate measures; Model-brain representational similarity analysis
Methods, data and reproducibility
Online and in-person listeners judged whether a category was present in 1 to 5-source scenes, single undistorted sounds (balanced confusion design) or single sounds under 68 distortions (d'). Filterbank linear classifiers, CNNs and a self-supervised spectrogram transformer were trained or fine-tuned on 1.5 million synthetic scenes, scored on the same stimuli (AUC as d') and against auditory-cortex fMRI.
- Experiment 1a: online (Prolific) category detection in scenes of 1 to 5 sources · human · n = 196 analyzed of 250 recruited
- Experiment 1b: in-person replication with 2 to 5-source scenes played over a 133-speaker array · human · n = 8 participants
- Experiment 1c: online cueing variant, with the category named before the scene on half of trials · human · n = 108 analyzed of 131 recruited
- Experiment 2: online single-source detection balanced so every off-diagonal confusion cell is sampled · human · n = 200 analyzed of 246 recruited
- Experiment 3: online single-source detection under 68 distortions · human · n = 272 analyzed of 339 recruited; 3,468 conditions (68 distortions x 51 categories)
- GISE-51 isolated-sound recordings (51 categories) mixed into the EnvAudioScene training set of reverberant 1 to 5-source scenes · n = 16,357 clips; 1.5 million scenes
- AudioSet strongly labeled (time-indexed) subset used for pretraining
- Auditory-cortex fMRI responses to 165 two-second natural sounds (previously published dataset) · human · n = 8 participants; 7,694 voxels
- Methods & tools
- Prolific, GISE-51, AudioSet, VGGish, SSAST (self-supervised Audio Spectrogram Transformer), Hugging Face, PyTorch, Torchaudio, gammatone cochleagram model, Chi et al. (2005) spectrotemporal modulation filterbank, ridge regression voxelwise encoding, representational similarity analysis (RSA)
- Assay conditions & compute
- NVIDIA A100 GPUs on the Open Mind cluster at MIT; training took about four days.
- Reproducibility gaps
- The data and code repository is promised to go public at publication; for now it is shared through a Google Drive link intended for editors and reviewers. No software versions or environment files are given.
Limitations (6)
- The benchmark is a yes/no classification over a fixed set of 51 labels, so it misses the hierarchical, physical and non-verbal sides of how people recognize sounds. Matching it does not show that a model recognizes sounds as richly as humans do. · stated by the authors
- All models were trained or fine-tuned on scenes built from GISE-51 (16,357 clips, against over 2 million in AudioSet). Sources were mixed at random with every category equally likely, so the scenes lack real-world diversity, co-occurrence and frequency statistics; this may cap performance and alignment. · stated by the authors
- Exemplar-level comparisons are weighted toward categories with few unique exemplars, where models are plausibly undertrained, so they may not reflect model-human correspondence across the wider set of categories. · stated by the authors
- The models explain behavior as a consequence of optimization rather than through explicit mechanistic hypotheses, and their internal strategies were not probed. · stated by the authors
- The credit given to training-data scale is confounded: the three best models also differ in architecture and input (log-mel inputs for VGGishPretrained and SSASTPretrained, a 0.96 s crop for VGGishPretrained), and SSASTPretrained is the largest network. · noted in this summary
- The link between behavioral and brain alignment rests on a qualitative trend across a small set of models with no reported statistic, using one previously published fMRI dataset of 8 participants. · noted in this summary
Who should read:Read this if you build or evaluate audio classifiers as models of human hearing and want a large human benchmark of sound recognition, covering multi-source scenes, confusions and distortions, to test them against.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Robust brain inspired prediction of speech recognition accuracy across many words, voices, and natural background sounds
Alex C Clonan, Ian H. Stevenson, Monty A. Escabı́·bioRxiv·Oct 9, 2026
- Deep neural network models of sound localization reveal how perception is adapted to real-world environments
Andrew F. Francl, Josh H. McDermott·Nature Human BehaviourPeer-reviewed·Jan 27, 2022
- Interference of mid-level sound statistics underlie human speech recognition sensitivity in natural noise
Alex C Clonan, Xiu Zhai, … Monty A. Escabı́·bioRxiv·Feb 13, 2024
- Causal inference in environmental sound recognition
J. Traer, S. V. Norman-Haignere, J. H. McDermott·CognitionPeer-reviewed·Sep 1, 2021
- Investigating Invariances in Auditory Event Categorization with Model Metamers
Hee So Kim, Elizabeth J. Lee, … Jenelle Feather·Conference on Cognitive Computational Neuroscience (CCN)Peer-reviewed·Jan 1, 2026
- Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions
Greta Tuckute, Jenelle Feather, … Josh H. McDermott·PLoS BiologyPeer-reviewed·Dec 13, 2023