Mind Captioning: Evolving descriptive text of mental content from human brain activity

Abstract

A central challenge in neuroscience is decoding brain activity to uncover mental content comprising multiple components and their interactions. Despite progress in decoding language-related information from human brain activity, generating comprehensive descriptions of complex mental content associated with structured visual semantics remains challenging. We present a method that generates descriptive text mirroring brain representations via semantic features computed by a deep language model. Constructing linear decoding models to translate brain activity induced by videos into semantic features of corresponding captions, we optimized candidate descriptions by aligning their features with brain-decoded features through word replacement and interpolation. This process yielded well-structured descriptions faithfully capturing viewed content, even without relying on the canonical language network, thereby revealing explicit representations of fine-grained structured semantic information outside this network. The method also successfully generalized to verbalize recalled content, demonstrating the potential for non-verbal thought-based brain-to-text communication, which could aid individuals with language expression difficulties.