bioRxivNeurosciencePreprint
GPT-2's activations predict the degree of semantic comprehension in the human brain
Charlotte CaucheteuxiD, Alexandre GramfortiD, Jean-Rémi KingiD
Abstract
Language transformers, like GPT-2, have demonstrated remarkable abilities to process text, and now constitute the backbone of deep translation, summarization and dialogue algorithms. However, whether these models encode information that relates to human comprehension remains controversial. Here, we show that the representations of GPT-2 not only map onto the brain responses to spoken stories, but also predict the extent to which subjects understand narratives. To this end, we analyze 101 subjects recorded with functional Magnetic Resonance Imaging while listening to 70 min of short stories. We then fit a linear model to predict brain activity from GPT-2s activations, and correlate this mapping with subjects comprehension scores as assessed for each story. The results show that GPT-2s brain predictions significantly correlate with semantic comprehension. These effects are bilaterally distributed in the language network and peak with a correlation of R=0.50 in the angular gyrus. Overall, this study paves the way to model narrative comprehension in the brain through the lens of modern language algorithms.
- organism: human
- assay: imaging
- research focus: benchmarking
- approach: transformer / foundation model
- approach: statistical model
- approach: mixed model
In-depth summary
Claude Opus 5.5 · xhigh effort · 7 Nov 2026 · from the full text of version 3 (2 Sept 2021) · CC BY 4.0
Across 101 listeners, how well GPT-2 layer-8 activations predict fMRI responses to spoken stories tracks story comprehension, with a correlation of R = 0.50 across subject-story pairs.
Earlier fMRI, MEG and intracranial studies cited here report that language-transformer activations linearly predict brain responses to text, but it is disputed whether these models capture anything related to human understanding, since they make trivial reasoning errors. This work asks whether the model-brain match varies with each listener's measured comprehension of a story, not only whether a mapping exists.
Key results
ShownListeners whose fMRI responses are better predicted from GPT-2 activations understood the story better, measured by subject-story pair.
Voxel-averaged brain score vs comprehension score: R = 0.50, p < 10^-15 (Pearson test, SciPy) across subject-story pairs (n = 237). The correlation is significant over bilateral temporal, parietal and prefrontal regions after FDR correction. The abstract reports a peak of R = 0.50 in the angular gyrus. · Abstract; GPT-2’s brain predictions correlate with…; Brain activations; Figure legends
ShownDifferences between stories do not explain the link between brain score and comprehension. It holds within each of the seven stories, and a mixed model finds no significant story-specific slope.
Linear mixed model with narrative as a random effect: fixed effect of brain score beta = 0.04, p < 10^-29; narrative-specific random slope beta < 10^-2, p > 0.11. Within single stories the correlation is above 0.40 for every story and reaches 0.76 for 'sherlock'. · Comprehension effects are mainly driven by…; Mixed-effect model; Replication across single narratives
SuggestedContextual GPT-2 representations relate to comprehension more strongly than low-level phonological features or GPT-2's non-contextual word embedding.
Brain scores from phonological features (word rate, phoneme rate, phonemes, stress, tone) correlate with comprehension at R = 0.17, p < 10^-2, which the paper gives as a difference of 0.32 below GPT-2. Averaged over voxels, the layer-0 word embedding gives a comprehension correlation 0.12 lower than layer 8. The paper reports no statistical test of these differences. · Low-level processing only partially accounts for…; The reliability of high-level representations…; Phonological features
SuggestedComprehension effects separate by processing level across regions. Phonological effects peak in left superior temporal cortex; lexical effects in superior temporal lobe and pars triangularis; contextual effects in superior frontal cortex, posterior superior temporal gyrus, precuneus and inferior frontal gyrus.
Region-wise correlations of comprehension with three scores: phonological-feature brain scores, word-embedding minus phonological scores, and layer-8 minus word-embedding scores. Regions come from a Destrieux subdivision with 142 regions per hemisphere, FDR-corrected (Benjamini/Hochberg) over 2 x 142 regions. Peaks are described qualitatively, with no values in the text. · Low-level processing only partially accounts for…; The reliability of high-level representations…; Significance; Brain parcellation
ShownGPT-2 activations predict fMRI over a bilateral cortical network. Middle layers fit best, and layer 8 is used for all later analyses.
Brain scores are significant (two-sided Wilcoxon test across subject-story pairs, FDR-corrected), peaking in the middle and superior temporal gyri and sulci, supramarginal cortex and inferior frontal cortex. Of layers 1 to 12, layer 8 fits best, and conclusions are reported unchanged for layers 6 to 12. The text gives no brain-score magnitudes. · GPT-2’s activations linearly map onto fMRI…; GPT-2 activations; Significance
Methods, data and reproducibility
For 237 subject-story pairs, ridge regression with a five-delay finite impulse response model predicted voxelwise fMRI from GPT-2 activations. The cross-validated Pearson correlation (the brain score) was then correlated with each listener's end-of-story comprehension score; phonological features and GPT-2's word embedding served as baselines, and a linear mixed model checked for story effects.
- Narratives fMRI dataset: brain responses of participants listening to spoken stories (full dataset) · human · n = 345 subjects, 27 narratives
- Analysed subset: seven stories with an end-of-story comprehension questionnaire whose scores varied across subjects (70 min of audio, 4 to 19 min per story) · human · n = 101 subjects; 237 (subject, narrative) pairs after excluding noisy recordings
- Methods & tools
- GPT-2 (pretrained, Huggingface transformers), GPT-2 tokenizer, scikit-learn (l2-penalized regression, cross-validation), SciPy (Pearson test), statsmodels (linear mixed-effects model), Gentle forced aligner, MNE, Destrieux atlas, functional MRI, Wilcoxon test with Benjamini/Hochberg FDR
- Reproducibility gaps
- No repository for the analysis code and no link or accession for the Narratives dataset are given. The footnotes link only to third-party software (Huggingface transformers, scikit-learn, Gentle, SciPy, MNE, statsmodels), with no versions.
Limitations (5)
- Brain scores are relatively low. The authors attribute this to averaging over all voxels to avoid selection bias, and to reporting without noise-ceiling correction, because pilot analyses showed the ceiling depends strongly on how it is computed. · stated by the authors
- The correlation between comprehension and GPT-2 mapping (R = 0.50) is far from perfect, so GPT-2-based modelling of brain responses does not account for all variation in comprehension. · stated by the authors
- The findings are correlational. Causal factors such as prior knowledge, attention and language complexity were not manipulated and are left to future work. · stated by the authors
- Only one language model (GPT-2) is tested, against phonological and word-embedding baselines. No other language models are compared. · noted in this summary
- The methods are internally inconsistent: the mapping section names layer nine while the rest of the paper uses layer eight, and the region count is 314 in the results but 2 x 142 in the methods and figure legends. · noted in this summary
Read this if you build encoding models linking language-model activations to fMRI and want an example that relates model-brain fit to a behavioural measure (story comprehension) across individual listeners.
- Citations
- 45
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training
Eghbal A. Hosseini, Martin Schrimpf, … Evelina G. Fedorenko·Neurobiology of LanguagePeer-reviewed·Jan 1, 2024
- A hierarchy of linguistic predictions during natural language comprehension
Micha Heilbron, Kristijan Armeni, … Floris P. de Lange·Proceedings of the National Academy of SciencesPeer-reviewed·Aug 3, 2022
- Driving and suppressing the human language network using large language models
Greta Tuckute, Aalok Sathe, … Evelina G. Fedorenko·Nature Human BehaviourPeer-reviewed·Jan 3, 2024
- Correspondence between the layered structure of deep language models and temporal structure of natural language processing in the human brain
Ariel Goldstein, Eric Ham, … Uri Hasson·bioRxiv·Jul 11, 2022
- Attention weights accurately predict language representations in the brain
Mathis Lamarre, Catherine Chen, Fatma Deniz·Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)Peer-reviewed·Jan 1, 2022
- Prediction, Syntax and Semantic Grounding in the Brain and Large Language Models
Nikola Kölbl, Stefan Rampp, … Achim Schilling·Scientific ReportsPeer-reviewed·Mar 10, 2026