Query-driven generative AI synthesizes multi-modal spatial omics from histology

Abstract

Spatial omics technologies offer unprecedented insights into the cellular organization of tissues; however, they are not yet scalable for routine clinical use. In contrast, images of histological staining remain the foundation of pathological diagnosis despite lacking molecular information. Bridging this gap requires computational methods that can accurately infer spatial molecular data from histology alone. Here, we introduce TissueCraftAI, a generative artificial intelligence framework that predicts multi-modal spatial omics maps directly from standard histology images using natural language prompts. To train and validate our model, we created PRISM-12M, a large-scale dataset comprising over twelve million spatially registered histology and spatial omics image patches across fourteen tissue types from humans and mice. TissueCraftAI significantly outperforms existing methods in generating realistic histology images and predicting spatial proteomics and transcriptomics data with high fidelity. We demonstrated its utility in various downstream applications, including improving cell type annotation and enhancing the accuracy of patient survival predictions across multiple cancer types. By enabling flexible, query-driven in silico spatial molecular analysis using routine histology images, TissueCraftAI opens up new research avenues in computational pathology.