Improving Emotion Classification by Combining fNIRS-Derived Hemodynamic Responses with Peripheral Physiological Signals

Abstract

We investigated whether multimodal sensing that combines functional near-infrared spectroscopy (fNIRS) with peripheral physiological signals can improve subject-independent classification of arousal and valence, the fundamental affective dimensions in Russells circumplex model. We developed Japanese emotion-inducing music-video stimuli (60 seconds each) and recorded subjects central nervous system activity using fNIRS, alongside peripheral physiological measures, specifically electrodermal activity (EDA) and photoplethysmography (PPG), during video viewing. To prioritize reproducibility and methodological transparency, we extracted simple, easily computed features from each modality and performed binary (high vs. low) classification separately for arousal and valence using a support vector machine. The combination of fNIRS and EDA yielded the highest performance, with a macro-averaged F1 score of 0.73 for arousal and 0.64 for valence. These findings underscore the utility of integrating fNIRS with peripheral physiological signals for subject-independent emotion classification.