Hierarchical Uncertainty Transformation During Multimodal Comprehension

Abstract

Human communication requires combining information from different modalities even when each source is incomplete or ambiguous. Yet how uncertainty is coordinated as distinct semantic representations interact and converge remains unclear. Here we propose a hierarchical uncertainty transformation framework and test it across four gesture-speech experiments combining behavioral measures, information-theoretic modelling, and electroencephalography. Rather than being uniformly reduced as information accumulates, uncertainty changed its computational form across three interdependent levels: uncertainty within modality-specific semantic representations, bidirectional but asymmetric changes during cross-modal interaction, and uncertainty associated with multisensory convergence. These computational distinctions were accompanied by differentiated but temporally overlapping neural associations. Modality-specific uncertainty was expressed relatively early, whereas cross-modal transformation and convergence were most prominently associated with activity in the N400 range. Critically, both the neural expression and statistical coupling of these computational quantities varied with the semantic information available from gesture and speech. Together, these findings show that multimodal comprehension involves a context-dependent reorganization of uncertainty across interacting representations, providing a computational framework for understanding how coherent meaning emerges from heterogeneous and uncertain information.