Language-aligned models and structured scene descriptions reveal sensitivity to compositional scene structure in the high-level visual cortex

Abstract

Natural scenes are defined not only by their contained objects, but also by the structured relations among those objects. Using 7T fMRI data from the Natural Scenes Dataset, we examined which information in scene-description representation contributes to the prediction of high-level visual cortical responses. We compared intact narratives with shuffled full description (bag-of-words), extracted content-word lists, and descriptions regenerated from those lists without access to the images of original narratives. Intact narrative descriptions outperformed the tested lexical controls. Both original and regenerated descriptions provided incremental predictive gains beyond bag-of-words, but the regenerated gain was only 21.3% as large as the original gain. Thus, constrained sentence reconstruction provided some additional predictive information without reproducing the original-description advantage. Exploratory image ratings indicated that this prediction advantage was greatest for scenes with richer scene compositional organization. Further, a language-aligned vision model provided additional predictive information beyond a self-supervised vision-only model, with a cortical distribution resembling the narrative advantage. These results address the representational basis of caption-based neural predictivity and are consistent with a contribution of structured scene information beyond simple entity co-occurrence. They further imply that language supervision may help induce comparable structure-sensitive representations in artificial vision models.