A transformation from vision to imagery in the human brain

Abstract

Extensive work has shown that the visual cortex is reactivated during mental imagery, and that models trained on visual data can predict imagery activity and decode imagined stimuli. These findings may explain why imagery can feel and function like vision, but give little insight into how the brain activity patterns that encode seen and mental images differ. While one popular theory ("weak vision") suggests that imagery differs from vision only in strength, recent work points to more complex differences. To clarify the relationship between visual and imagery activity in the human brain, we introduce the concept of an imagery transformation--a mapping from visual to imagery activity patterns evoked by the same stimulus. Importantly, this approach can describe a variety of possible scenarios, from simple rescaling to the selective removal or reorientation of activity dimensions. Using two 7T fMRI datasets, we estimated imagery transformations across different visual areas and found they accurately predict imagery activity. We show that imagery transformations are indeed more complex than simple weakening: in early visual cortex, they halve the number of active dimensions and reorient them, such that reconstructions of visual activity in terms of imagery dimensions explain only 25-50% of the variance. Nonetheless, these reoriented dimensions still coarsely encode the features encoded by the principal dimensions of visual activity. These findings help to explain the "same but different" relationship between mental imagery and vision: imagery activity patterns are transformations of visual activity patterns that approximate them, but encode fewer features and occupy a distinct subspace within the overall space of brain activations.