Unsupervised alignment reveals structural commonalities and differences in neural representations of natural scenes across individuals and brain areas

Abstract

Neuroscience research has extensively explored the commonality of neural representations of sensory stimuli across individuals to uncover universal neural mechanisms in the encoding of sensory information. To compare neural representations across different brains, Representational Similarity Analysis (RSA) has been used, which focuses on the similarity structures of neural representations for different stimuli. Despite the broad applicability and utility of RSA, one limitation is that its conventional framework assumes that neural representations of particular stimuli correspond directly to those of the same stimuli in different brains. This assumption excludes the possibility that neural representations correspond differently and limits the exploration of finer structural similarities. To overcome this limitation, we propose to use an unsupervised alignment framework based on Gromov-Wasserstein Optimal Transport (GWOT) to compare similarity structures without presupposing stimulus correspondences. This method allows for the identification of optimal correspondence between neural representations of stimuli based solely on internal neural representation relationships, and thereby provides a more detailed comparison of neural similarity structures across individuals. We applied this unsupervised alignment to investigate the commonality of representational similarity structures of natural scenes, using large datasets of Neuropixels recordings in mice and fMRI recordings in humans. We found that the similarity structure of neural representations in the same visual cortical areas can be well aligned across individuals in an unsupervised manner in both mice and humans. In contrast, we found that the degree of alignment across different brain areas cannot be fully explained by proximity in the visual processing hierarchy alone, but also found some reasonable alignment results, such that the similarity structures of higher-order visual areas can be well aligned with each other but not with lower-order visual areas. We expect that our unsupervised approach will be useful for revealing more detailed structural commonalities or differences that may not be captured by the conventional supervised approach.