Resolving the treachery of images: Neural signatures during natural interactions with real objects are as reliable as while viewing images

Abstract

Is a real object the same as its image? Vision and its neural basis have been studied almost entirely with images. However, real objects are fundamentally different, because we can act on them. But since we see and interact with real objects differently each time, it is widely believed that brain activity will be highly variable. An alternative view is that real objects should evoke reliable neural signatures since brain activity might be optimized for natural behavior. To investigate these possibilities, we performed wireless neural recordings from the high-level visual (IT) and motor regions (PMv/PFC) of freely moving monkeys, performing a natural task where they ate food items offered by humans, and a controlled task where they viewed the corresponding images on a screen. Neural decoders trained on images accurately identified the corresponding real objects. Even though neural responses were more variable during real object interactions, the variability of object-relevant information was strikingly similar, suggesting that the additional variability is present along irrelevant dimensions. Finally, key events in the natural task elicited reliable neural signatures. We conclude that brain activity is efficiently organized during natural behavior, ensuring that real objects are encoded as efficiently as images.