Neocortical and hippocampal theta oscillations track audiovisual integration and replay of speech memories

Abstract

"Are you talkin to me?!" If you ever watched the masterpiece "Taxi driver" directed by Martin Scorsese, you certainly recall the famous monologue during which Travis Bickle rehearses an imaginary confrontation in front of a mirror. While remembering this scene, you recollect a myriad of speech features across visual and auditory senses with a smooth sensation of unified memory. The aim of this study was to investigate how brain oscillations integrate the fine-grained synchrony between coinciding visual and auditory features when forming multisensory speech memories. We developed a memory task presenting participants with short synchronous or asynchronous movie clips focusing on the face of speakers engaged in real interviews. In the synchronous condition, the natural alignment between visual and auditory onsets was kept intact. In the asynchronous condition, auditory onsets were delayed to present lip movements and speech sounds in antiphase specifically with respect to the theta oscillation synchronising them in the original movie. We recorded magnetoencephalographic (MEG) activity to investigate brain oscillations in response to audiovisual asynchrony in the theta band. Our results first showed that theta oscillations in the neocortex and hippocampus were modulated by the level of synchrony between lip movements and syllables during audiovisual speech perception. Second, the accuracy of subsequent theta oscillation reinstatement during memory recollection was decreased when lip movements and the auditory envelope were encoded in asynchrony during speech perception. We demonstrate that neural theta oscillations in the neocortex and the hippocampus integrated lip movements and syllables during natural speech. We conclude that neural theta oscillations play a pivotal role in both aspects of audiovisual speech memories, i.e., encoding and retrieval.