Biomimetic computations improve neural network robustness

Abstract

Object recognition by natural and artificial sensory systems requires a combination of selectivity and invariance. Both natural and artificial neural networks achieve selectivity and invariance by propagating sensory information though layers of neurons organised in a functional hierarchy. Both employ computational units performing AND-like operations for selectivity and OR-like operations for invariance. However, while biological neurons are intrinsically capable of switching between these operations, their artificial counterparts are hard-wired to perform only one of them. We wanted to test whether the flexible mapping between neurons and computations observed in biological neural networks is compatible with, or perhaps even useful to, artificial neural networks. To answer this question, we have developed a deep learning layer in which both selectivity and invariance operations can be performed by the same neurons. As with biological neurons, the choice of which operation an artificial neuron performs on a given input can be governed by the input strength. This flexible layer successfully outputs a combination of the two operations and, surprisingly, confers additional robustness to adversarial examples, which are inputs deliberately crafted to promote misclassification. The flexible mapping also improves accuracy when the training dataset is small, as well as when data are corrupted by certain types of noise. These results narrow the gap between biological and artificial neural networks and add a new bio-inspired approach to the arsenal of defenses against adversarial examples, which are known threats to model-based optimization and network security. Author summaryThe biophysical properties of a biological neuron enable it to perform both OR-like and AND-like operations over its inputs. In contrast, artificial neural networks use units that always perform only one specific operation. We wondered whether the flexibility observed in individual biological neurons could be incorporated into artificial neural networks without breaking them. If artificial neural networks required their units to perform single operations, then this requirement would either point to a fundamental difference between biological and artificial neural networks or indicate that biological neurons are somehow constrained to perform only one type of operations during object recognition. If, on the other hand, artificial units can flexibly switch between different operations, then such a result would indicate that biological and artificial neural networks are more similar than previously thought and would be an important step towards biomimetic AI. To find out, we introduced a new computational structure we call a flexible layer, in which individual units can switch between operations according to a rule (e.g., depending on input strength, or even randomly). We found that inserting the flexible layer in one or several positions in different artificial neural networks trained on several benchmark datasets not only preserves their accuracy but also makes them more robust to various perturbations and improves learning when training data is scarce.