When knowledge interferes with perception: Neural mechanisms of the semantic amplification of visual false memory

Abstract

Visual false memory refers to our tendency to falsely recognize novel stimuli that are visually similar to seen stimuli. Visual false memory also occurs when stimuli are meaningful, suggesting that semantic information interferes with the encoding of visual details. However, the neural mechanisms of this semantic interference effect are largely unknown. In the present fMRI study, participants were scanned while encoding visually similar fonts presented with words (word-fonts) or pseudowords (pseudoword-fonts), and later, when recognizing old, new similar (lures), and new dissimilar (novel) fonts displayed in the same meaningless letter string. We performed (1) representational similarity analysis (RSA) at encoding to identify visual, visuosemantic, and semantic representations associated with subsequent visual true and false font recognition, (2) encoding-retrieval similarity (ERS) analysis to assess their reinstatement during retrieval, and (3) mediational analyses to examine hippocampal contributions. The study yielded three main findings. First, visuosemantic representations supported true font recognition when stored in right fusiform gyrus, but false recognition of word-fonts when stored in the left fusiform gyrus. Second, mirroring this pattern, reinstatement in right fusiform gyrus was associated with true font recognition, whereas reinstatement in left fusiform gyrus was linked to false recognition of word-fonts. Finally, posterior hippocampal activation reduced false font memory mainly for pseudoword-associated fonts via decreased reinstatement in perceptual regions, while anterior hippocampal activity increased false memory of word-fonts via enhanced reinstatement in semantic regions. Taken together, these findings reveal how distinct hippocampal-cortical pathways differentially bias memory towards perceptual specificity or semantic generalization. Significance StatementFalse memories are often triggered by visual similarity, but this study shows that meaning encoded during learning can distort memory for visual details, even when retrieval cues are meaningless. Participants learned fonts associated with words or pseudowords and judged whether similar lure fonts, shown on a meaningless letter string, were seen before. Although behavioral performance was similar across conditions, brain imaging revealed a key dissociation: the left fusiform gyrus and anterior hippocampus promote semantic generalization that increases false recognition, whereas the right fusiform gyrus and posterior hippocampus support perceptual specificity that protects against it. These findings reveal how distinct hippocampal-cortical pathways differentially bias memory toward truth or illusion.