A Multivariate Comparison of EEG and fMRI to ECoG Using Visual Object Representations in Humans

1.Today most neurocognitive studies in humans employ the non-invasive neuroimaging techniques fMRI and EEG. However, how the data provided by fMRI and EEG relate exactly to the underlying neural activity remains incompletely understood. Here, we pursued this open question by relating EEG and fMRI data at the level of neural population codes using multivariate pattern analysis. For this, we recorded EEG and fMRI data while participants viewed everyday objects in different viewing conditions, and then related the data to ECoG data recorded for the same stimulus set. The comparison of EEG and ECoG data showed that object category signals emerge swiftly in the visual system and can be detected by both EEG and ECoG at similar temporal delays after stimulus onset. The correlation between EEG and ECoG reduces and onset latencies increase when object representations tolerant to changes in scale and orientation are considered. The comparison of fMRI and ECoG overall revealed a tighter relationship in occipital than in temporal regions, related to differences in fMRI signal-to-noise ratio. Together, our results reveal a complex relationship between fMRI, EEG and ECoG signals at the level of population codes that critically depends on the time point after stimulus onset, the region investigated, and the visual contents used.