Situation Models in the Brain are Used to Resolve Word References

Abstract

Resolving the meaning of a pronoun requires retrieving information about a unique individual (a referent) from many remembered experiences and inferences throughout sensory and cognitive modalities, from vision and audition to episodic memory and social cognition. Here, we hypothesize that the functional neuroanatomy of pronoun interpretation involves interaction across distributed neural networks, during which each referent activates its own unique sensorimotor neural fingerprint associated with these experiences. To test this hypothesis, we collected fMRI data from 20 people watching a full-length movie, and developed a 3D branched convolutional neural network to distinguish movie characters from the fMRI signal across distributed sensorimotor regions. The same model distinguished the characters referenced by pronouns, using these same sensorimotor regions, supplemented by the hippocampus, precuneus, and medial prefrontal cortices. This work has far-reaching implications for understanding the relations among the many domains and modalities of neural representation required for ecological language comprehension. In particular, the demonstration that situation models, implemented in distributed sensory-motor and association cortices, are involved in resolving reference, suggests a whole brain distribution for language processing.