henhaobio
Sign in

bioRxivNeurosciencePreprint

Language processing in brains and deep neural networks: computational convergence and its limits

Charlotte CaucheteuxiD, Jean-Rémi KingiD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

AO_SCPLOWBSTRACTC_SCPLOWDeep Learning has recently led to major advances in natural language processing. Do these models process sentences similarly to humans, and is this similarity driven by specific principles? Using a variety of artificial neural networks, trained on image classification, word embedding, or language modeling, we evaluate whether their architectural and functional properties lead them to generate activations linearly comparable to those of 102 human brains measured with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We show that image, word and contextualized word embeddings separate the hierarchical levels of language processing in the brain. Critically, we compare 3,600 embeddings in their ability to linearly map onto these brain responses. The results show that (1) the position of the layer in the network and (2) the ability of the network to accurately predict words from context are the main factors responsible for the emergence of brain-like representations in artificial neural networks. Together, these results show how perceptual, lexical and compositional representations precisely unfold within each cortical region and contribute to uncovering the governing principles of language processing in brains and algorithms.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (14 Jan 2021) · CC BY 4.0

In 3,600 transformer checkpoints compared with fMRI and MEG of 102 readers, middle layers best predict brain activity and grow more brain-like as word-prediction accuracy improves; input embeddings diverge.

Earlier studies linking language models to brain recordings used a few heterogeneous pretrained models differing in size, architecture, objective and corpus, and random embeddings can already correlate with brain activity, so it was unclear whether training actually drives convergence. Here many transformers are trained from scratch on one corpus and tracked across training, separating the contributions of layer position, architecture, task and language-modeling accuracy to brain similarity.

Key results

  1. ShownAcross transformer variants and both training objectives, middle layers predict fMRI and MEG responses better than input or output layers, and relative layer position accounts for most of the variation in brain scores.

    Middle vs output layers: fMRI ΔR = 0.011 ± 0.001, MEG ΔR = 0.003 ± 0.000; middle vs input: fMRI ΔR = 0.031 ± 0.001, MEG ΔR = 0.009 ± 0.001. Random-forest permutation importance: layer position explained 81.5 ± 1.2% of fMRI and 70.2 ± 1.4% of MEG scores, with architecture and task under 17%. The effect held across 32 architectures and both causal and masked objectives. · 2.3 The middle layers of trained language models…; 4.4.2 Feature importance

  2. ShownDuring training, middle-layer brain scores rise with word-prediction accuracy. Input-layer (word-embedding) scores improve early, then decline even though the networks keep getting better at their training task.

    Middle layer brain score vs top-1 accuracy: MEG R = 0.87 ± 0.01, fMRI R = 0.82 ± 0.02. Input-layer scores rose over the first 100K iterations (≈200M words); after that, training step correlated negatively with brain score (MEG R = −0.166 ± 0.019, fMRI R = −0.106 ± 0.021, p < 10^−4). Each network was frozen at about 100 log-spaced steps up to 4.5M updates. · 2.4 Only middle layers converge towards brain…; 4.1.2 Language Transformers; 4.4.1 Convergence analysis

  3. ShownAmong model properties, language-modeling accuracy is the strongest predictor of brain similarity. Training task, layer width, number of attention heads and depth have only modest effects.

    Random-forest feature importance (drop in R when one feature is shuffled): accuracy ΔR = 0.56 ± 0.02 (fMRI) and 0.51 ± 0.02 (MEG); amount of training 0.15 ± 0.01 / 0.21 ± 0.01; relative layer position 0.47 ± 0.02 / 0.19 ± 0.01; training task, dimensionality, heads and total layers ΔR < .08. · 2.4 Only middle layers converge towards brain…; 4.4.2 Feature importance

  4. ShownVisual, word-embedding and middle-layer transformer features map onto a spatiotemporal reading hierarchy, running from V1 at about 100 ms to temporal, frontal and parietal language areas.

    Character-recognition CNN peaked in V1: fMRI R = 0.022 ± 0.003, MEG R = 0.008 ± 0.002 at 100 ms. Word embeddings from a 13-layer, 128-dimensional transformer peaked near 400 ms in left temporal (R = 0.052 ± 0.004) and frontal cortex (R = 0.053 ± 0.003). Middle-layer gains over word embeddings: superior temporal ΔR = 0.012, angular 0.010, infero-frontal 0.016, dorsolateral prefrontal 0.012 (each ± 0.001). · 2.2 Image, word and compositional embeddings…; 4.1.1 Visual Convolutional Neural Network

  5. ShownNoise ceilings (each subject predicted from the others) are much lower for MEG than for fMRI, especially in source space. Most encoding scores are very low, which the authors attribute to single-sample encoding, but they fall within the noise-ceiling range.

    Each subject was predicted from the other subjects who read the same sentence. fMRI ceiling averaged R = 0.129 ± 0.004 (SEM) in the superior temporal gyrus; MEG ceilings peaked at R = 0.069 ± 0.001. · 2.1 Average and single-trial fMRI and MEG…; 3 Discussion; 4.3 Noise Ceiling: Brain → Brain mapping

Methods, data and reproducibility

Cross-validated ridge-regression encoding models mapped activations from a character-recognition VGG CNN and from Wikipedia-trained transformers (causal or masked language modeling, about 100 checkpoints each) onto fMRI and source-localized MEG responses to 400 sentences. Random-forest permutation importance related model properties to brain scores.

  • fMRI (Siemens Trio 3T) and MEG (CTF) recordings of native Dutch right-handed speakers reading sentences one word at a time (Schoffelen and colleagues; Donders Institute) · human · n = 102 subjects; 100 in fMRI and 92 in MEG analyses after exclusions
  • Sentence stimuli: meaningful sentences presented word by word in sequences of 9 to 15 words · n = 400 distinct sentences, 120 per subject
  • Wikipedia training corpus for all transformers (extracted with WikiExtractor, Moses-tokenized, 50,341-word vocabulary) · n = 278,386,651 words
  • Held-out Wikipedia test set for top-1 word-prediction accuracy · n = 180,883 words
Methods & tools
VGG convolutional network (word-image recognition), XLM transformer implementation, WikiExtractor, Moses tokenizer, FreeSurfer (recon-all), fMRIPrep, nilearn, MNE-Python, scikit-learn (GroupKFold, RandomForest), PALS Brodmann area atlas, Destrieux atlas, CTF MEG and Siemens Trio 3T MRI
Assay conditions & compute
Each transformer was trained on 8 GPUs with early stopping; GPU model, training time and memory are not stated. For computational reasons, MEG model comparisons used ten time samples between 0 and 2 s.
Reproducibility gaps
No code repository for the analyses or trained models is given; the only printed link is to the third-party WikiExtractor tool. The neuroimaging data are attributed to Schoffelen and colleagues and the Donders Institute without an accession or access route. Software versions not stated.
Limitations (5)
  • Most encoding correlations are very low. The authors attribute this to single-sample encoding, note that the scores fall within the range of the noise ceilings, and stress that signal-to-noise ratio is a major limitation of neuroimaging. · stated by the authors
  • The authors call the convergence partial. Transformers are feedforward, attend over a very large word buffer and minimize prediction error only at the output, unlike recurrent predictive-coding accounts of the brain, and they are trained on far more text than a person reads. · stated by the authors
  • Each contextualized word is modeled independently, and representations are only coarsely split into three levels (visual, lexical, compositional). · stated by the authors
  • All language models are small transformers (4 to 12 layers, 128 to 512 dimensions) trained from scratch on one Wikipedia corpus, and all brain data come from a single isolated-sentence reading dataset. Larger pretrained models and other cohorts or languages are not tested. · noted in this summary
  • Counts are inconsistent across the text: 102 vs 100 subjects, 92 vs 95 MEG subjects, two vs three training tasks (Figure 1), 32 architectures vs the factor grid listed, and a 13-layer model in Section 2.2 vs 4 to 12 layers elsewhere. · noted in this summary

Read this if you use language-model activations as encoding models for fMRI or MEG and want evidence on which layers and which model properties drive similarity to brain responses.

Citations
49

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISGoldstein '22Goldstein '22Wingfield '22Eichert '26Varda '26Beguš '23
77% semantic matchNeuroscience
Ariel Goldstein, Eric Ham, Samuel A. Nastase+16 more·bioRxiv·Jul 11, 2022· 22 citations