Characteristic sounds facilitate object search in real-life scenes

Abstract

Real-world multisensory events do not only provide temporally and spatially correlated information, but also semantic correspondences about object identity. Semantically consistent sounds can enhance visual detection, identification and search performance, but these effects are always demonstrated in simple and stereotyped displays that lack ecological validity. In order to address identity-based crossmodal relationships in real world scenarios, we designed a visual search task using complex, dynamic scenes. Participants searched objects in video clips from real life scenes with background sounds. Auditory cues embedded in the background sounds could be target-consistent, distracter-consistent, neutral and no sound (just background noise). We found that characteristic sounds enhance visual search of relevant objects in natural scenes but fail to increase the salience of irrelevant distracters. Our findings generalize previous results on object-based crossmodal interactions with simple stimuli and shed light upon how audio-visual semantically congruent relationships play out in real life contexts.