Eye and hand coarticulation during problem solving reveals hierarchically organized planning

During everyday activities--such as preparing a cup of coffee or traveling across cities--we often plan ahead and execute sequences of actions. However, much remains to be understood about how we plan and coordinate such sequences (e.g., eye and hand movements) to solve novel and challenging tasks, for which plans must be formed from scratch. This study investigates how participants coordinate gaze and cursor movements during problem solving tasks that involve selecting a trajectory on a grid connecting multiple targets. By focusing on the action execution phase, we aimed to probe the structure of the gaze-cursor plans that participants used to solve the tasks. Our analysis reveals three main findings. First, consistent with previous studies, participants segment the problem into sequences of gestures; within each gesture, gaze focuses on a target and remains fixed until the cursor reaches it, then shifts to the next target. Second, both gaze position--while fixating on the current target--and the kinematics of cursor movement leading up to the current target allow prediction of the next cursor movements direction, revealing coarticulation in both cursor-cursor and gaze-cursor movements. Third, and most interestingly, the position of the gaze around the current target aligns with the direction of the next saccade, revealing coarticulation between successive gaze fixations. Together, these findings show that participants break the problem into gesture sequences and plan multiple eye and cursor movements in advance to efficiently reach both the current and upcoming gesture targets. This suggests a hierarchical planning strategy, with participants planning ahead at two levels: gesture targets and cursor movements.