CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps

Abstract

Recent advances in brain decoding work has shown the potential of Vision Transformers (ViTs) to learn mappings from neural representation to visual features. Here we present CortexDecoder, a brain decoding framework mapping fMRI flatmaps to images viewed by subjects. Flatmaps are neural responses projected onto cortical surfaces and then unfolded into 2D images. We demonstrate that (1) flatmaps preserve sufficient signal to achieve high performance in brain decoding; raw fMRI data has noisy signals and high-dimensionality which makes decoding a difficult task. (2) Reshaping raw fMRI data into flatmaps allows for clean patch-based tokenization. This allows us to harness state-of-the-art Vision Transformers to learn meaningful neural representations and preserve spatial encoding. To do this, we pass flatmap patches into a pretrained CortexMAE encoder to obtain brain tokens. A lightweight cross-attention bridge then learns to map these brain tokens to image features. Our cross-attention bridge is split into two modules; the semantic module for predicting CLIP embeddings and the lower-level module to generate low-level image reconstruction. Finally, a fine-tuned SDXL (Stable Diffusion XL) takes this low-level reconstruction image as the initial image. Conditioned by the CLIP semantic embedding, the SDXL produces the final reconstruction image. In single-subject evaluations, CortexDecoder achieves performance comparable to state-of-the-art brain decoding methods, demonstrating the effectiveness of flatmap based neural representations for visual reconstruction.