Temporal coding enables hyperacuity in event based vision

Abstract

The fact that the eyes are constantly in motion, even during fixation, entails that the spike times of retinal outputs carry information about the visual scene even when the scene is static. Moreover, this motion implies that fine details of the visual scene could not be decoded from pure spatial retinal representations due to smearing. Understanding the interplay of temporal and spatial information in visual processing is thus pivotal for both biological research and bio-inspired computer-vision applications. In this study, we consider data from a popular event-based camera that was designed to emulate the function of a biological retina in hardware. Similarly to biological eye, and in contrast to standard frame-based cameras, this camera outputs an asynchronous sequence of "spike" events. We used this camera to obtain dataset of event streams of tiny images, i.e., images whose recognition is impaired by photosensors pixelization and thus their recognition requires hyperacuity. Using these datasets we demonstrate here the superiority of event-based spatio-temporal coding over frame-based spatial coding in the recognition of tiny images by artificial neural networks (ANNs). We further demonstrate the benefits of event sequences for unsupervised learning. Interestingly, Vernier hyperacuity, which is a standard measure of shape hyperacuity, emerged in ANNs following training on tiny images, resembling the natural hyperacuity observed in humans. Our findings underscore the essential role of precise temporal information in visual processing, offering insights for advancing both biological understanding and bio-inspired engineering of visual perception.