The representation and valuation of subgoals in the human brain during model-based hierarchical behavior

Abstract

The human capacity to plan and perform long, complex sequences of behavior to achieve distant goals depends in part on a hierarchical organization that divides behavior into structured segments. Such a mechanism requires the internal designation of certain states as subgoals to mark the successful implementation of a behavioral segment. How the brain represents subgoals over time and computes decision values as a function of subgoals is unknown. While most characterizations of hierarchical behavior lack knowledge of the environment, human decision-making also relies on planning with an internal model of the world. Consequently, it remains to be determined how the brain computes values of subgoals using model-based planning in order to drive hierarchical, model-based decisions. Using a sequential-subgoal decision-making task designed to evoke hierarchical, model-based behavior in combination with fMRI, we decoded a representation of the current subgoal in insula and ventromedial prefrontal cortex during decision-making that persisted over time-a critical, latent representation for computing values and orienting behavior in the correct sequence. Using a model-based, hierarchical reinforcement learning model, we also found key decision signals based on values from the model in several regions of frontal cortex. These findings thereby shed light on the neural correlates of subgoal representation and illustrate how value signals can be computed on the basis of these subgoals and knowledge of the environment structure.