Neural Networks and Foundation Models: Two Strategies for EEG-to-fMRI Prediction

Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) are two widely used neuroimaging techniques, with complementary strengths and weaknesses. Predicting fMRI activity from EEG activity could give us the best of both worlds, and open new horizons for neuroscience research and neurotechnology applications. Here, we formulate this prediction objective both as a classification task (predicting whether the fMRI signal increases or de- creases) and a regression task (predicting the value of this signal). We follow two distinct strategies: training classical machine learning and deep learning mod- els (including MLP, CNN, RNN, and transformer) on an EEG-fMRI dataset, or leveraging the capabilities of pre-trained large language models (LLMs) and large multimodal models. We show that predicting fMRI activity from EEG activity is possible for the brain regions defined by the Harvard-Oxford cortical atlas, in the context of subjects performing a neurofeedback task. Interestingly, both strategies yield promising results, possibly highlighting two complementary paths for our prediction objective. Furthermore, a Chain-of-Thought approach demonstrates that LLMs can infer the cognitive functions associated with EEG data, and subsequently predict the fMRI data from these cognitive functions. The natural combination of the two strategies, i.e., fine-tuning an LLM on an EEG-fMRI dataset, is not straightforward and would certainly require further study. These findings could provide important insights for enhancing neural interfaces and advancing toward a multimodal foundation model for neuroscience, integrating EEG, fMRI, and possibly other neuroimaging modalities.