Prediction, Syntax and Semantic Grounding in the Brain and Large Language Models

Abstract

Language comprehension involves continuous prediction of upcoming words, with syntactic structure and semantic meaning intertwined in the human brain. To date, few studies have used combined magnetoencephalography (MEG) and electroen-cephalography (EEG) measurements to investigate how syntactic processing, predictive coding, and semantic grounding interact in real time. Here we present the first combined MEG-EEG investigation of syntactic processing and semantic grounding under naturalistic conditions. Twenty-nine healthy participants listened to a German audio book while their neural responses were recorded. Event-related fields and event-related potentials for four word classes - nouns, verbs, adjectives, and proper nouns - showed highly reproducible, characteristic spatio-temporal signatures, including significant pre-onset activity for nouns, suggesting enhanced predictability of this word class. Source-space analyses revealed pronounced activation in the pre- and post-central gyri for nouns, suggesting a deeper semantic grounding of nouns in e.g. sensory experiences than verbs. To further investigate predictive mechanisms, we analyzed the hidden representations of the large language model Llama. By comparing the transformer-based representations to neural responses, we explored the relationship between computational language models and human brain activity, offering new insights into syntactic and semantic prediction. These findings highlight the power of simultaneous MEG-EEG recordings in unraveling the predictive, syntactic, and semantic mechanisms that underlie the comprehension of natural language.