Pupil- and gaze dynamics track emotion content in naturalistic speech

Abstract

Cortical tracking of speech features is a well-established marker of continuous speech processing, but far less is known about listeners ocular responses to speech rhythm. Ocular responses are central to active sensing models in the auditory domain, where motor recruitment guides temporal speech prediction and attention allocation, potentially shaped by non-rhythmic cues, such as emotions. Here, we ask whether listeners pupil response and eye movements track the acoustic speech signals and to what extent this tracking is modulated by emotion-related top-down factors, including subjective emotion ratings, mood, and trait empathy. In a validation study (N = 100), participants passively listened to two TED talks and intermittently rated segments on valence (negative-positive) and arousal (low-high). This suggested substantial variability in valence and arousal across speech segments in both talks. In the second study (N = 41), participants completed the same task while pupillometry and electrooculography (EOG) were recorded. Mutual information was used to quantify speech tracking in pupil dilation, along with horizontal and vertical eye movements. All ocular signals significantly tracked speech at low frequencies. High-arousal speech was associated with stronger pupil tracking but weaker vertical and horizontal EOG tracking. Negative speech valence was linked to stronger tracking in pupil and vertical eye-movement signals. Interactions between the speech-emotion dimensions, as well as their interactions with listeners mood, further shaped these effects, giving rise to distinct patterns across ocular measures. Taken together, our findings provide evidence that ocular activity dynamically aligns to the temporal structure of natural speech and that this tracking is sensitive to both stimulus-driven and listener-dependent emotional factors.