Understanding transformation tolerant visual object representations in the human brain and convolutional neural networks

Abstract

Forming transformation-tolerant object representations is critical to high-level primate vision. Despite its significance, many details of tolerance in the human brain remain unknown. Likewise, despite the ability of convolutional neural networks (CNNs) to exhibit human-like object categorization performance, whether CNNs form tolerance similar to that of the human brain is unknown. Here we provide the first comprehensive documentation and comparison of three tolerance measures in the human brain and CNNs. We measured fMRI responses from human ventral visual areas to real-world objects across both Euclidean and non-Euclidean feature changes. In single fMRI voxels in higher visual areas, we observed robust object response rank-order preservation across feature changes. This is indicative of functional smoothness in tolerance at the fMRI meso-scale level that has never been reported before. At the voxel population level, we found highly consistent object representational structure across feature changes towards the end of ventral processing. Rank-order preservation, consistency, and a third tolerance measure, cross-decoding success (i.e., a linear classifiers ability to generalize performance across feature changes) showed an overall tight coupling. These tolerance measures were lower for Euclidean than non-Euclidean feature changes in lower visual areas, but increased over the course of ventral processing in most cases. These characteristics of tolerance, however, were absent in eight CNNs pretrained with ImageNet images with varying network architecture, depth, the presence/absence of recurrent processing, or whether a network was pretrained with the original or stylized ImageNet images that encouraged shape processing. Most notably, CNNs do not show increased representational consistency across feature changes at the higher layers. CNNs thus do not appear to develop the same kind of tolerance as the human brain over the course of visual processing. Significant StatementPerceiving object identity among changes in non-identity features and forming transformation-tolerant object representations is essential to high-level primate vision. Here we provide a comprehensive documentation and comparison of three tolerance measures between the human brain and CNNs pretrained for object classification. While all three measures show increased tolerance in the human brain across four types of feature changes towards the end of ventral visual processing, CNNs fail to develop the same kind of tolerance with visual processing.