Towards a unified theory of the reference frame of the ventriloquism aftereffect

Abstract

The ventriloquism aftereffect (VAE), observed as a shift in the perceived locations of sounds after audio-visual stimulation, requires reference frame alignment since hearing and vision encode space in different reference frames (head-centered vs. eye-centered). Previous experimental studies reported inconsistent results, observing either a mixture of head-centered and eye-centered frames, or a predominantly head-centered frame. Here, a computational model is introduced to examine these inconsistencies. Based on experimental data, the model uses the measured size of the ventriloquism effect to predict the VAE adaptation in the auditory spatial map. Versions of the model examine whether the adaptation is induced by visual signals in head-centered frame, eye-centered frame, by eye-gaze direction-dependent signals, or their combination, and whether some biases are induced by the saccade-to-auditory-target response method used in the experiments. The model is first evaluated on three separate data sets. It can predict them well even without explicit need for an eye-centered signals influencing VAE, suggesting that the reference frame of VAE is mainly head-centered. The model predictions are qualitatively similar but less accurate when all three data sets are combined, suggesting that interactions between individual neural mechanisms are more complex than the simple linear combination assumed in the model.