The processing of semantic complexity and co-speech gestures in schizophrenia: a naturalistic, multimodal fMRI study

Schizophrenia is marked by aberrant processing of complex speech and gesture, which may contribute functionally to its impaired social communication. To date, extant neuroscientific studies of schizophrenia have largely investigated dysfunctional speech and gesture in isolation, and no prior research has examined how the two communicative channels may interact in more natural contexts. Here, we tested if patients with schizophrenia show aberrant neural processing of semantically complex story segments, and if speech-associated gestures (co-speech gestures) might modulate this effect. In a functional MRI study, we presented to 34 participants (16 patients and 18 matched-controls) an ecologically-valid retelling of a continuous story, performed via speech and spontaneous gestures. We split the entire story into ten-word segments, and measured the semantic complexity for each segment with idea density, a linguistic measure that is commonly used clinically to evaluate aberrant language dysfunction at semantic level. Per segment, the presence of numbers of gestures varied (n = 0, 1, +2). Our results suggest that, in comparison to controls, patients showed reduced activation for more complex segments in the bilateral middle frontal and inferior parietal regions. Importantly, this neural aberrance was reduced in segments presented with gestures. Thus, for the first time with a naturalistic multimodal stimulation paradigm, we show that gestures reduced group differences when processing a natural story, probably by facilitating the processing of semantically complex segments of the story in schizophrenia.