Multimodal deep neural decoding of visual object representation in humans

Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. However, it is unclear how neural activities in spatially distributed brain regions, especially in terms of temporal dynamics, represent visual objects. To address this issue, we explored the spatial and temporal organization of visual object representations using concurrent functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). Visualization of the fMRI DNN revealed that visual categorization (faces or non-face objects) occurred in brain-wide cortical regions, including the ventral temporal cortex. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner.