Cross-Modal Organization of Facial and Vocal Behavior Revealed by Coordination Metrics Derived from Temporal Alignment

Abstract

Facial and vocal behaviors jointly convey affective and communicative information, yet their cross-modal organization remains difficult to characterize quantitatively. We propose a framework for characterizing facial-vocal organization by aligning facial and vocal dynamics using canonical time warping. Multivariate facial action unit trajectories and vocal prosodic trajectories were extracted from recordings of spontaneous speech and aligned in a shared latent space. From the alignment process, we derived three cross-modal coordination metrics: "scaling factor", reflecting the relative scaling between modalities; "time shift ratio", reflecting the frequency of temporal shifts; and "time shift magnitude", reflecting their size. We evaluated these metrics in verbally fluent adults with autism spectrum disorder (ASD; N = 35) and controls (N = 44). The proposed metrics did not differ significantly between diagnostic groups but showed associations with individual differences in communication and anxiety. Scaling factor was linked to trait anxiety in ASD, whereas time shift ratio was associated with communication characteristics across participants. The proposed metrics also showed little association with conventional vocal pitch variability, suggesting that they capture aspects of expressive organization not reflected by vocal variability alone. Our findings support alignment-derived metrics as a complementary approach for characterizing individual variability in facial-vocal expressive behavior.