Modelling Audio-Visual Reaction Time with Recurrent Mean-Field Networks

Abstract

Audio-visual stimuli are typically detected faster and more accurately than unisensory ones, but it remains unclear whether this benefit reflects true integration of evidence or other effects such as independent processing and modality repetition. We address this with three biologically motivated models of audio-visual integration, built on Wong and Wang model, a mean-field recurrent framework originally developed for unisensory decision-making. The OR model takes in independent unisensory processes through winner-take-all competition, and the SUM model implements full linear summation of sensory evidence. These two sit at opposite ends of the integration spectrum. The REPEAT model adds switch and repeat costs, whereby responses are faster when the modality repeats across trials. We simulated 121 participants whose unisensory evidence accumulation rates varied from modality dominance to balanced integration. We fitted the simulated reaction times and accuracy with a drift diffusion model and compared the resulting multisensory drift rates with the optimal prediction from the unisensory rates, as well as with empirical audio-visual detection data. The OR and SUM Models reproduced established unisensory response patterns from behavioural data from in children, children with autism, adults, older adults, and people with Parkinson's disease. Drift diffusion analysis revealed suboptimal integration in the OR Model and optimal integration in the SUM Model. However, the SUM Model also produced supra-optimal responses under certain conditions, inconsistent with behavioural data. The REPEAT Model captured the role of priming in sensory repetition effects, distinguishing it from true multisensory integration. Together, these models highlight how biologically grounded mathematical frameworks can shed light on the mechanisms underlying multisensory integration, particularly the nuanced contributions of modality repetition and integration efficiency.