OmniT2I: a unified table-to-image framework for survival prediction and cross-modal inference with incomplete multimodal cancer data

Abstract

Molecular profiles and histopathology provide complementary information about cancer prognosis, but these data are often incomplete for individual patients. Here we present OmniT2I, a table-to-image framework for survival prediction and cross-modal inference from incomplete multimodal cancer data. Each molecular profile was converted into a three-channel image representation with correlation-based feature placement. OmniT2I combines available molecular and histopathology embeddings through availability-aware fusion and predicts missing modalities from those observed. Across 10,373 patients with 32 cancer types, the C-index was 0.631 within cancer types and 0.783 with all cancers pooled (0.660 and 0.787 in 5,605 patients with all five modalities). A single model handled all 31 modality combinations and, compared with nine survival methods trained separately for each, achieved higher C-indices in 240 of 279 pooled and 226 of 279 within-cancer comparisons. Adding predictions of withheld modalities to observed inputs improved pooled C-indices in all 180 pairings of observed and withheld modality sets and within-cancer C-indices in 164. Adding measured histopathology to RNA produced the largest within-cancer C-index gain among the four additional modalities evaluated, whereas adding copy number variation reduced it. In seven external cohorts, frozen RNA-only C-indices differed from test performance in the corresponding cancer type in TCGA by a median of -0.003 (range, -0.092 to +0.079). OmniT2I enables prognostic modeling with incomplete multimodal data.