Hist2Cell: Deciphering Fine-grained Cellular Architectures from Histology Images

Abstract

Histology images, with low cost, are unleashing great power of predicting cellular phenotypes in tissue, thanks to the emerging spatial transcriptomics serving as annotations. Recent efforts aimed to predict individual gene expression, suffering from low accuracy and high variability, while no methods are tailored to predict fine-grained transcriptional cell types -the most critical phenotype. Here, we present Hist2Cell, a Vision Graph-Transformer framework, to accurately resolve fine-grained transcriptional cell types (up to 40 cell types) directly from histology images and further create cellular maps of diverse tissues at a customizable resolution. Specifically, trained on human lung and breast cancer spatial transcriptome datasets, Hist2Cell accurately predicts the abundance of each cell type across space in new patient samples with Pearson Correlation Coefficient of biological informative cell types over 0.80, and effectively capturing their colocalization directly from histology images. Moreover, without re-training, it robustly generalizes to large-scale histology cancer cohorts from TCGA, highlighting recurrent cell co-localization and supporting precise survival prediction by revealing distinct tissue micro-environments and insightful cell type-patient mortality relationship. Therefore, Hist2Cell enables cost-efficient histology analysis for large-scale studies of spatial biology and precise cancer prognosis in real-world clinical diagnostics.