A performance-optimized model of neural responses across the ventral visual stream

Abstract

Human visual object recognition is subserved by a multitude of cortical areas. To make sense of this system, one line of research focused on response properties of primary visual cortex neurons and developed theoretical models of a set of canonical computations such as convolution, thresholding, exponentiating and normalization that could be hierarchically repeated to give rise to more complex representations. Another line or research focused on response properties of high-level visual cortex and linked these to semantic categories useful for object recognition. Here, we hypothesized that the panoply of visual representations in the human ventral stream may be understood as emergent properties of a system constrained both by simple canonical computations and by top-level, object recognition functionality in a single unified framework (Yamins et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Guclu and van Gerven, 2015). We built a deep convolutional neural network model optimized for object recognition and compared representations at various model levels using representational similarity analysis to human functional imaging responses elicited from viewing hundreds of image stimuli. Neural network layers developed representations that corresponded in a hierarchical consistent fashion to visual areas from V1 to LOC. This correspondence increased with optimization of the models recognition performance. These findings support a unified view of the ventral stream in which representations from the earliest to the latest stages can be understood as being built from basic computations inspired by modeling of early visual cortex shaped by optimization for high-level object-based performance constraints.\n\nSignificance StatementPrior work has taken two complimentary approaches to understanding the cortical processes underlying our ability to visually recognize objects. One approach identified canonical computations from primary visual cortex that could be hierarchically repeated and give rise to complex representations. Another approach linked later visual area responses to semantic categories useful for object recognition. Here we combined both approaches by optimizing a deep convolution neural network based on canonical computations to preform object recognition. We found that this network developed hierarchically similar response properties to those of visual areas we measured using functional imaging. Thus, we show that object-based performance optimization results in predictive models that not only share similarity with late visual areas, but also intermediate and early visual areas.