Two cortical mechanisms for natural audiovisual processing

Understanding how the human brain processes natural audiovisual information remains a central challenge in cognitive neuroscience. Progress has been limited by the difficulty of modeling complex audiovisual stimuli; prior work has therefore relied largely on short, controlled, or unimodal stimuli, leaving cortical mechanisms supporting real-world comprehension poorly characterized. Although recent AI advances enable extraction of high-dimensional, time-resolved features from naturalistic stimuli, how cortical regions dynamically prioritize auditory and visual information as time unfolds remains largely unexplored. Using large-scale fMRI data collected during movie viewing, we developed two complementary prediction-based approaches to map moment-by-moment sensory processing across cortex: one detects sustained periods when one modality predicts a region better than the other, identifying regions that switch the modality they encode for meaningful stretches of time; while the other identifies periods when both modalities predict the region well, revealing balanced audiovisual representations. Together, these analyses reveal two cortical patterns of audiovisual processing: two modality-switching ``bows" (a posterior bow encircling category-selective visual cortex and an anterior bow spanning dorsolateral frontal cortex) and an arrow-like axis jointly representing both modalities (extending from lateral occipital into temporal cortex). We further show that switching is not driven by reduced low-level information in the non-dominant modality, but by reduced cortical encoding of that information, suggesting active sensory reweighting. Modality dominance also cannot be explained simply by the dominant modality conveying a region's preferred concepts; switching regions do not merely track whichever modality expresses these concepts.Together, these systems suggest a cortical architecture that adaptively reweights sensory inputs while maintaining balanced multimodal representations during naturalistic comprehension.