Heterogeneity in Neuronal Dynamics is Learned by Gradient Descent for Temporal Processing Tasks

Individual neurons in the brain have complex intrinsic dynamics that are highly diverse. We hypothesize that the complex dynamics produced by networks of complex and heterogeneous neurons may contribute to the brains ability to process and respond to temporally complex data. To study the role of complex and heterogeneous neuronal dynamics in network computation, we develop a rate-based neuronal model, the generalized-leaky-integrate-and-firing-rate (GLIFR) model, which is a rate-equivalent of the generalized-leaky-integrate-and-fire model. The GLIFR model has multiple dynamical mechanisms which add to the complexity of its activity while maintaining differentiability. We focus on the role of after-spike currents, currents induced or modulated by neuronal spikes, in producing rich temporal dynamics. We use machine learning techniques to learn both synaptic weights and parameters underlying intrinsic dynamics to solve temporal tasks. The GLIFR model allows us to use standard gradient descent techniques rather than surrogate gradient descent, which has been utilized in spiking neural networks. After establishing the ability to optimize parameters using gradient descent in single neurons, we ask how networks of GLIFR neurons learn and perform on temporally challenging tasks, such as sinusoidal pattern generation and sequential MNIST. We find that these networks learn a diversity of parameters, which gives rise to diversity in neuronal dynamics. We also observe that training networks on the sequential MNIST task leads to formation of cell classes based on the clustering of neuronal parameters. GLIFR networks have mixed performance when compared to vanilla recurrent neural networks but appear to be more robust to random silencing. When we explore these performance gains further, we find that both the ability to learn heterogeneity and the presence of after-spike currents contribute. Our work both demonstrates the computational robustness of neuronal complexity and diversity in networks and demonstrates a feasible method of training such models using exact gradients.