Thinking ahead: prediction in context as a keystone of language in humans and machines
Ariel GoldsteiniD, Zaid ZadaiD, Eliav Buchnik, Mariano Schain, Amy Rose PriceiD, Bobbi Aubrey, Samuel A. NastaseiD, Amir FederiD, Dotan Emanuel, Alon CoheniD, Aren Jansen, Harshvardhan GazulaiD, Gina ChoeiD, Aditi Rao, Se Catherine Kim, Colton CastoiD, Lora FandaiD, Werner DoyleiD, Daniel J. FriedmaniD, Patricia C. DuganiD, Lucía MelloniiD, Roi ReichartiD, Sasha DevoreiD, Adeen FlinkeriD, Liat HasenfratziD, , Uri HassoniD
Abstract
Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models are trained to generate appropriate linguistic responses in a given context. We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process natural language: 1) both are engaged in continuous next-word prediction before word-onset; 2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise (i.e., prediction error signals); 3) both represent words as a function of the previous context. In support of these three principles, our findings indicate that: a) the neural activity before word-onset contains context-dependent predictive information about forthcoming words, even hundreds of milliseconds before the words are perceived; b) the neural activity after word-onset reflects the surprise level and prediction error; and c) autoregressive DLM contextual embeddings capture the neural representation of context-specific word meaning better than arbitrary or static semantic embeddings. Together, our findings suggest that autoregressive DLMs provide a novel and biologically feasible computational framework for studying the neural basis of language.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 51
- Organism
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 4 (30 Sept 2021) · CC BY 4.0
In nine epilepsy patients, cortical activity carries next-word information up to 800 ms before onset, tracks GPT-2 surprise after onset, and fits GPT-2 contextual embeddings better than static ones.
Evidence that listeners predict words before hearing them has been indirect (cloze tests on controlled sentences, stronger responses 400 ms after unexpected words), and earlier brain studies using language models did not treat autoregressive models as cognitive models. This work adds a word-by-word prediction protocol for natural speech and intracranial evidence of spontaneous pre-onset prediction, linked to GPT-2 surprise and contextual embeddings.
Key results
ShownPatients were never asked to predict, yet their left-hemisphere cortical activity carried information about the identity of the upcoming word hundreds of milliseconds before they heard it. This held even with arbitrary embeddings that carry no relations between words.
In 160 left-hemisphere electrodes selected with GloVe, linear encoding was above chance up to 800 ms before onset (peak 150-200 ms after); shuffled GloVe gave flat encoding near zero. Arbitrary embeddings also encoded pre-onset, holding after removing the first word of every bigram that repeats in the story, and a previous-plus-current-word model differed from a previous-word-only model pre-onset. · Localizing neural responses to natural speech…; Encoding neural responses before word onset; Encoding neural responses before word onset…
ShownListeners guessed upcoming words in an unfamiliar natural story well above a most-frequent-word baseline. Word-level predictability for humans and GPT-2 closely agreed, and agreement grew as GPT-2 was given longer context.
Mean predictability 28% (SE 0.5%) vs 6% for always guessing 'the'; human vs GPT-2 predictability r = .79 (p < .001); top-1 guesses matched for 49.1% of words, with 27.6% jointly correct and 54.7% jointly wrong. Correlation rose from r = .46 with two-word context to an asymptote of r = .79 at 100 words; 2- to 5-gram models agreed with humans less than GPT-2 did. · Word-by-word behavioral prediction during a…; Comparing human and DLM next-word predictions…; Prediction as a function of contextual window…; Figure legends
ShownBefore onset, activity encoded the word GPT-2 predicted even when that prediction was wrong. After onset, it encoded the word actually heard, which separates anticipation from processing of the perceived word.
Words were split by whether they fell in GPT-2's top-5 predictions (62% counted correct). Pre-onset encoding with GloVe vectors of the predicted word was strong whether the prediction was right or wrong; post-onset encoding followed the perceived word. Splitting by listeners' most-predicted words gave similar results; a GPT-2 top-1 split is also reported (Fig. S8). · Neural signals before word onset contain…; Accuracy split; Figure legends
SuggestedWords GPT-2 failed to predict drew stronger encoding and activity about 400 ms after onset, more in the inferior frontal gyrus (IFG) than the middle superior temporal gyrus (mSTG). GPT-2 entropy tracked pre-onset activity and cross-entropy tracked post-onset activity.
Accuracy x region ANOVA at the 400 ms lag (23 IFG, 28 mSTG electrodes): interaction F(1,49) = 8.42, p = .006. Across 160 electrodes, activity rose pre-onset for correctly predicted words and about 400 ms post-onset for mispredicted ones; partial correlations linked GPT-2 entropy negatively to pre-onset activity and cross-entropy to post-onset activity, peaking near 400 ms. · Increased semantic processing for unpredictable…; Increased activity as a function of surprise…
ShownGPT-2 contextual embeddings beat static embeddings for both encoding and decoding. The gain depended on the context of each specific occurrence: averaging a word's contextual vectors removed it, and shuffling them across occurrences reduced it.
GPT-2 encoding was significant in 208 left-hemisphere electrodes (34 right), 71 of them missed by GloVe. Averaging each word's vectors cut encoding to GloVe level; shuffling across occurrences reduced it substantially. Decoder AUC 0.74 (GPT-2) vs 0.68 (GloVe) and 0.68 (arbitrary) at the 150 ms lag; GPT-2 decoding detected word information from about -1000 ms. · Localizing neural responses to natural speech…; Representing word meaning in unique contexts; Using contextual embeddings for predicting the…
Methods, data and reproducibility
Six groups of 50 online participants predicted every word of a 30-minute podcast via a sliding 10-word window; their guesses were compared with GPT-2. Electrocorticography power from nine epilepsy patients hearing the podcast was modelled per electrode and lag by linear encoding from arbitrary, GloVe and GPT-2 embeddings, and decoded by a neural network scored on word classification.
- Sliding-window next-word guesses collected on Mechanical Turk for the 30-minute 'Monkey in the Middle' story (This American Life podcast); 5113 words, 5078 after a technical error · human · n = 300 participants (six groups of 50)
- Electrocorticography from patients with refractory epilepsy at the NYU School of Medicine Comprehensive Epilepsy Center, freely listening to the same story · human · n = 10 recorded, 9 analysed; 1339 electrodes (1106 left, 233 right)
- Brown corpus (NLTK built-in) used to train 2- to 5-gram baseline models
- Methods & tools
- GPT-2 (pre-trained, Hugging Face), GloVe, word2vec, NLTK, Penn Phonetics Lab Forced Aligner, FieldTrip, MATLAB 2019a, Electrode Localization Toolbox for MATLAB, Mechanical Turk, electrocorticography (ECoG), NicoletOne C64 and Neuroworks Quantum amplifiers, MNI152 template
- Data & accessions
- http://www.nltk.org/nltk_data/
- Software & environment
- MATLAB 2019a with FieldTrip for preprocessing; pre-trained GPT-2 via Hugging Face; NLTK for n-gram baselines. No other software versions, containers or requirement files are given.
- Reproducibility gaps
- No repository for the authors' own analysis code and no deposit of the behavioral or ECoG data are given. The two printed links are a third-party electrode-localization toolbox and the NLTK corpus page.
Limitations (6)
- The links between GPT-2 entropy or surprise and neural signals are correlational. The authors leave two questions to future work: whether there is a causal connection, and whether the 400 ms post-onset increase reflects surprise estimation, extra attention or semantic processing. · stated by the authors
- Shared computational principles do not mean a shared implementation. The authors note that transformers parallelize what the brain does word by word, so transformers are not biologically feasible. · stated by the authors
- Right-hemisphere coverage was much sparser (233 vs 1106 electrodes), so the analyses focus on the left hemisphere and cannot test lateralization. · stated by the authors
- Wavelet power estimation and smoothing let up to about 93 ms of future signal leak into each time point. The authors judge this small relative to the pre- versus post-onset effects. · stated by the authors
- All neural results come from one 30-minute story and nine epilepsy patients whose electrode placement was set clinically. No second stimulus or cohort tests whether the results replicate. · noted in this summary
- GPT-2 is the only autoregressive model tested, so conclusions about autoregressive language models as a class rest on a single model. · noted in this summary
Who should read:Read this if you model neural or behavioral responses to natural speech with language-model embeddings or surprisal and want controlled evidence that cortex carries next-word information before onset.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- A hierarchy of linguistic predictions during natural language comprehension
Micha Heilbron, Kristijan Armeni, … Floris P. de Lange·Proceedings of the National Academy of SciencesPeer-reviewed·Aug 3, 2022
- Correspondence between the layered structure of deep language models and temporal structure of natural language processing in the human brain
Ariel Goldstein, Eric Ham, … Uri Hasson·bioRxiv·Jul 11, 2022
- Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain
Ariel Goldstein, Avigail Grinstein-Dabush, … Uri Hasson·bioRxiv·Mar 1, 2022
- A unifying computational account of temporal context effects in language across the human cortex
Vy A. Vo, Shailee Jain, … Alexander G. Huth·bioRxiv·Aug 3, 2023
- Stimulus dependencies---rather than next-word prediction---can explain pre-onset brain encoding in naturalistic listening designs
Inés Schönmann, Jakub Szewczyk, … Micha Heilbron·bioRxiv·Mar 8, 2025
- The neural architecture of language: Integrative modeling converges on predictive processing
Martin Schrimpf, Idan Blank, … Evelina G. Fedorenko·Proceedings of the National Academy of SciencesPeer-reviewed·Nov 4, 2021