Linguistic Context Shapes Representations in the Human Visual Cortex

Abstract

Language provides a way to anticipate visual scenes before they are seen, but how this context relates to visual representations across levels of abstraction remains unclear. Here, 210 hours of functional magnetic resonance imaging from eight participants performing a delayed caption-image matching task are analyzed using computational encoding, decoding, and alignment models. Caption-evoked activity carries semantic information across visual cortex, including early visual areas. Subsequent image responses vary with both the strength of caption-derived contextual representations and the discrepancy between anticipated and viewed scenes, assessed across lower-level visual and higher-level semantic features. Two functionally distinct voxel populations are identified within early visual cortex: suppression populations respond less strongly to matching than mismatching images, whereas enhancement populations show the opposite pattern. Despite their lower responses to matching images, suppression populations support comparable or better retrieval of corresponding caption-evoked responses using fewer voxels. The two populations also differ in their temporal dynamics, representational structure, and relationship to behavior. Together, these findings link linguistic context to both the strength and informational content of visual cortical responses, revealing how language-vision relationships are expressed across representational levels and two functionally distinct populations in early visual cortex.