Priority-based transformations of stimulus representation in visual working memory

Abstract

How does the brain prioritize among the contents of working memory (WM) to appropriately guide behavior? Using inverted encoding modeling (IEM), previous work (Wan et al., 2020) showed that unprioritized memory items (UMI) are actively represented in the brain but in a "flipped", or opposite, format compared to prioritized memory items (PMI). To gain insight into the mechanisms underlying the UMI-to-PMI representational transformation, we trained recurrent neural networks (RNNs) with an LSTM (long short-term memory) architecture to perform a 2-back working memory task. Although visualization of LSTM hidden layer activity using Principal Component Analysis (PCA) suggested that stimulus representations undergo a smooth rotational transformation across the trial, demixed (d)PCA of the same data decomposed this pattern into a cascade of multiple trajectories, each with a different time course, unfolding within UMI and PMI subspaces. The application of the same analyses to the EEG dataset of Wan et al. (2020) indicated that an items trajectory through the UMI subspace closely mirrored that of the RNN, but that its trajectory through the PMI subspace differed markedly from the RNN. It may be a general principle that, at the level of the representational code, information held in WM undergoes priority-based transformations that allow for its retention while preventing it from interfering with concurrent behavior. Implementational details of this process may vary across model systems. Author SummaryHow is information held in working memory (WM) but outside the current focus of attention? Motivated by previous neuroimaging studies, we trained recurrent neural networks (RNNs) to perform a 2-back WM task that entails shifts of an items priority status. Dimensionality reduction of the resultant activity in the hidden layer of the RNN allowed us to characterize how a stimulus items representation follows a transformational trajectory through high-dimensional representational space as its priority status changes from memory probe to unprioritized to prioritized. This work illustrates the value of artificial neural networks for assessing and refining hypotheses about mechanisms for information processing in the brain.