Loss of muscarinic M5 receptor function manifests disparate impairments in exploratory behavior in male and female mice despite common dopamine regulation

There are five cloned muscarinic acetylcholine receptors (M1-M5). Of these, the muscarinic type 5 receptor (M5) is the only one localized to dopamine neurons in the ventral tegmental area and substantia nigra. Unlike M1-M4, the M5 receptor has relatively restricted expression in the brain, making it an attractive therapeutic target. Here we performed an in-depth characterization of M5-dependent potentiation of dopamine transmission in the nucleus accumbens and accompanying exploratory behaviors in male and female mice. We show that M5 receptors potentiate dopamine transmission by acting directly on the terminals within the nucleus accumbens. Using the agonist oxotremorine, we revealed a unique concentration response curve and a sensitivity to repeated stressor exposure. We found that constitutive deletion of M5 receptors reduced exploration of the center of an open field while at the same time impairing normal habituation only in male mice. In addition, M5 deletion reduced exploration of salient stimuli, especially under conditions of high novelty, yet had no effect on hedonia. We conclude that M5 receptors are critical for both engaging with the environment and updating behavioral output in responses to the environment cues, specifically in male mice. A cardinal feature of mood and anxiety disorders is a withdrawal from the environment. These data indicate that boosting M5 receptor activity may be a useful therapeutic target for ameliorating these symptoms of depression and anxiety. Significance StatementThe basic physiological and behavioral functions of the muscarinic M5 receptor remain understudied. Furthermore, its presence on dopamine neurons, relatively restricted expression in the brain, and recent crystallization make it an attractive target for therapeutic development. Yet, most preclinical studies of M5 receptor function have primarily focused on substance use disorders in male rodents. Here we characterized the role of M5 receptors in potentiating dopamine transmission in the nucleus accumbens, finding impaired functioning after stress exposure. Furthermore, we show that M5 receptors can modulate exploratory behavior in a sex-specific manner, without impacting hedonic behavior. These findings further illustrate the therapeutic potential of the M5 receptor, warranting further research in the context of treating mood disorders.