Where do I go? Decoding temporal neural dynamics of scene processing and visuospatial memory interactions using CNNs

Abstract

Visual scene perception enables rapid interpretation of the surrounding environment by integrating multiple visual features related to task demands and context, which is essential for goal-directed behavior. In the present work, we investigated the temporal neural dynamics underlying the interaction between the processing of visual features (i.e., bottom-up processes) and contextual knowledge (i.e., top-down processes) during scene perception. We analyzed EEG data from 30 participants performing scene memory and visuospatial memory tasks in which we manipulated the number of navigational affordances available (i.e., the number of open doors) while controlling for similar low-level visual features across tasks. We used convolutional neural networks (CNN) coupled with gradient-weighted class activation mapping (Grad-CAM) to assess the main channels and time points underlying neural processing for each task. We found that early occipitoparietal activity (50-250 ms post-stimulus) contributed most to the classification of several aspects of visual perception, including scene color, navigational affordances, and spatial memory content. In addition, we showed that the CNN successfully trained to detect affordances during scene perception was unable to detect the same affordances in the spatial memory task after learning, whereas a similarly trained and tested model for detecting wall color was able to generalize across tasks. Taken together, these results reveal an early common window of integration for scene and visuospatial memory information, with a specific and immediate influence of newly acquired spatial knowledge on early neural correlates of scene perception.