Cross-task, explainable, and real-time decoding of human emotion states by integrating grey and white matter intracranial neural activity

Abstract

Decoding human emotion states from intracranial neural activity is key in developing brain-computer interfaces and new therapies for affective disorders. However, real-world application of decoding requires high performance that integrates neural activity from both grey and white matter, stable generalization across different contexts, sufficient neural encoding explainability, and robust real-time implementation, all of which remain elusive. Here, we simultaneously recorded intracranial electroencephalogram (iEEG) and abundant self-rated valence and arousal scores across two emotion-eliciting tasks in eighteen subjects, forming the largest intracranial neural dataset for emotion decoding. We then developed personalized decoding models within a hybrid deep learning framework involving both self-supervised and supervised components. The models achieved high-performance decoding of continuous valence/arousal states, doubling the R-squared performance of prior EEG/iEEG decoding. Critically, the models significantly improved performance by integrating grey and white matter signals and demonstrated cross-task generalization. The models further revealed shared and preferred mesolimbic-thalamo-cortical subnetworks encoding valence and arousal, showing neurophysiological explainability. Finally, the models realized robust real-time decoding in four new subjects. Our results have implications for advancing emotion decoding neurotechnology toward deployable affective brain-computer interfaces and closed-loop therapeutic systems for affective disorders.