Gesture's body orientation modulates the N400 during semantic integration of gesture and visual sentence

Abstract

Body orientation of gesture entails social-communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face-to-face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an EEG experiment presenting to participants (n=21) videos of frontal and lateral hand gestures of five-seconds (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the hand gesture (e.g., the mountain is high/low...). All participants underwent a semantic-probe task, judging whether a target word is related or unrelated to the gesture-speech event. EEG results suggest that, during the perception phase of hand-gestures, while both frontal and lateral gestures elicited power decrease in both the alpha (8-12Hz) and the beta (16-24Hz) bands, lateral gestures elicited reduced power decrease in the beta band when compared with frontal gestures. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture, frontal gestures elicited an N400 effect for gesture-sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. The findings suggest that body orientation plays a crucial role in gesture perception, and that its inferred social-communicative intention influences gesture-sentence semantic integration in an interactive manner.