Deep Multimodal Graph-Based Network for Survival Prediction from Highly Multiplexed Images and Patient Variables

Abstract

The spatial architecture of the tumour microenvironment and phenotypic heterogeneity of tumour cells have been shown to be associated with cancer prognosis and clinical outcomes, including survival. Recent advances in highly multiplexed imaging, including imaging mass cytometry (IMC), capture spatially resolved, high-dimensional maps that quantify dozens of disease-relevant biomarkers at single-cell resolution, that contain potential to inform patient-specific prognosis. However, existing automated methods for predicting survival typically do not leverage spatial phenotype information captured at the single-cell level, and current methods tend to focus on a single modality, such as patient variables (PVs). There is no end-to-end method designed to leverage the rich information in whole IMC images and all marker channels, and aggregate this information with PVs in a complementary manner to predict survival with enhanced accuracy. We introduce a deep multimodal graph-based network (DMGN) that integrates entire IMC images and multiple PVs for end-to-end survival prediction of breast cancer. We propose a multimodal graph-based module that considers relationships between spatial phenotype information in all image regions and all PVs, and scales each region-PV pair based on its relevance to survival. We propose another module to automatically generate embeddings specialised for each PV to enhance multimodal aggregation. We show that our modules are consistently effective at improving survival prediction performance using two public datasets, and that DMGN can be applied to an independent validation dataset across the same antigens but different antibody clones. Our DMGN outperformed state-of-the-art methods at survival prediction.