Neural affect decoding from minimal spectrotemporal glimpses of voice signals

Abstract

Decoding affect information from voice signals is challenging in natural environments where noise masks the most diagnostic and discriminative acoustic voice information. The extent to which neural affect decoding is resilient against or vulnerable to extreme noise-masking of voice signals remains largely unknown. Here, we performed three experiments using an auditory "bubble" technique to randomly mask and unmask certain spectrotemporal voice features similar to natural noise masking. First, using psychoacoustic voice-in-noise testing, we observed that a substantial part of the full spectrotemporal space of voice signals contains diagnostic cues essential for accurate affect decoding, contributing to the partial robustness of the voice signals under noise masking. Affective voice signals (joy, aggression) were correctly identified from early broadband spectral and temporally extended high-frequency information. Neutral voices were primarily identified from low- to medium-level spectral information with high perceptual relevance near the voice offset. Second, using human neuroimaging, we mechanistically linked neural activity to minimal spectrotemporal patches that enable accurate affect classification from voice samples. A neural network composed of (sub-)cortical auditory and limbic nodes, fronto-insular regions, and vocal motor mirroring nodes supported the affective discrimination of voice signals. More specifically, activity in the orbitofrontal cortex and amygdala was mechanistically linked to correct affect decisions, as determined by a reverse brain-to-decision mapping approach. Thus, decoding affect from voice signals embedded in complex noise seems partially feasible, provided that the noise does not obscure the minimal diagnostic spectrotemporal information essential for engaging cortical and subcortical limbic affect decoding mechanisms.