Cross-modal integration of reward value during oculomotor planning

Abstract

Reward value guides goal-directed behavior and modulates early sensory processing. Rewarding stimuli are often multisensory but it is not known how reward value is combined across sensory modalities. Here we show that the integration of reward value critically depends on whether the distinct sensory inputs are perceived to emanate from the same multisensory object. We systematically manipulated the congruency in monetary reward values and the relative spatial positions of co-occurring auditory and visual stimuli that served as bimodal distractors during an oculomotor task. The amount of interference induced by the distractors was used as an indicator of their perceptual salience. Our results across two experiments show that when reward value is linked to each modality separately, the value congruence between vision and audition determines the combined salience of the bimodal distractors. However, reward value of vision wins over the value of audition if visual and auditory stimuli have been experienced as belonging to the same audiovisual object prior to the learning of the reward values. The perceived spatial alignment of auditory and visual stimuli is a prerequisite for the integration of their reward values, as no effect of reward value was observed when the two modalities were perceived to be misaligned. These results show that in a task that highly relies on the processing of visual spatial information, the reward values from multiple sensory modalities are integrated with each other, each with their respective weights. This weighting depends on the congruency in reward values, exposure history, and spatial co-localization.