It takes neurons to understand neurons: Digital twins of visual cortex synthesize neural metamers

Abstract

Metamers, images that are perceived as equal, are a useful tool to study representations of natural images in biological and artificial vision systems. We synthesized metamers for the mouse visual system by inverting a deep encoding model to find an image that matched the observed neural activity to the original presented image. When testing the resulting images in physiological experiments we found that they most closely reproduced the neural activity of the original image when compared to other decoding methods, even when tested in a different animal whose neural activity was not used to produce the metamer. This demonstrates that deep encoding models do capture general characteristic properties of biological visual systems and can be used to define a meaningful perceptual loss for the visual system.