Capturing brain dynamics: latent spatiotemporal patterns predict stimuli and individual differences

Insights from functional Magnetic Resonance Imaging (fMRI), and more recently from recordings of large numbers of neurons through calcium imaging, reveal that many cognitive, emotional, and motor functions depend on the multivariate interactions of neuronal populations. To capture and characterize spatiotemporal properties of brain events, we propose an architecture based on long short-term memory (LSTM) networks to uncover distributed spatiotemporal signatures during dynamic experimental conditions1. We demonstrate the potential of the approach using naturalistic movie-watching fMRI data. We show that movie clips result in complex but distinct spatiotemporal patterns in brain data that can be classified using LSTMs ({approx} 90% for 15-way classification), demonstrating that learned representations generalized to unseen participants. LSTMs were also superior to existing methods in predicting behavior and personality traits of individuals. We propose a dimensionality reduction approach that uncovers low-dimensional trajectories and captures essential informational properties of brain dynamics. Finally, we employed saliency maps to characterize spatiotemporally-varying brain-region importance. The spatiotemporal saliency maps revealed dynamic but consistent changes in fMRI activation data. We believe our approach provides a powerful framework for visualizing, analyzing, and discovering dynamic spatially distributed brain representations during naturalistic conditions.