A Deep Reinforcement Learning based approach for movement training of neuro-musculoskeletal systems

Abstract

Mechanisms behind neural control of movement have been an active area of research. Goal-directed movement is a common experimental paradigm used to understand these mechanisms and relevant neural pathways. In this paper, we attempt to build an anatomically and physiologically realistic model of spinal cord along with the relevant circuitry and interface it with a musculoskeletal model of an upper limb, using the NEUROiD platform. The neuronal model (simulated on NEURON) and the musculoskeletal model (simulated on OpenSim) are cosimulated on NEUROiD. We then use Deep Reinforcement Learning to obtain a functionally equivalent model of the supraspinal components and the descending cortical activations feeding into the last-order interneurons and motoneurons. Uniplanar goal directed movement of the elbow joint was used as the goal for the learning algorithm. Key aspects of our work are: (1) Our solution converges naturally to the triphasic response observed in goal directed tasks (2) Gradually increasing the complexity of task helped in faster learning (3) In response to corticospinal inputs, our model could produce movements on which it was not explicitly trained, but were close to the trained movements. Being able to generate movements on which the model was not explicitly trained, implies that the movement repertoire that a biomimetic model needs to learn, could be much smaller than the complete set of movements it can execute. We hope that this will lead to building larger and complex biomimetic systems, one block at a time.