Adaptive stretching of representations across brain regions and deep learning model layers

Abstract

Prefrontal cortex (PFC) is known to modulate the visual system to favor goal-relevant information by accentuating task-relevant stimulus dimensions. Does the brain broadly re-configures itself to optimize performance by stretching visual representations along task-relevant dimensions? We considered a task that required monkeys to selectively attend on a trial-by-trial basis to one of two dimensions (color or motion direction) to make a decision. Except for V4 (color bound) and MT (motion bound), the brain radically re-configured itself to stretch representations along task-relevant dimensions in lateral PFC, frontal eye fields (FEF), lateral intraparietal cortex (LIP), and inferotemporal cortex (IT). Spike timing was crucial to this code. A deep learning model was trained on the same visual input and rewards as the monkeys. Despite lacking an explicit selective attention or other control mechanism, the model displayed task-relevant stretching as a consequence of error minimization, indicating that stretching is an adaptive strategy.