Synthetic whole-slide image tile generation with gene expression profiles infused deep generative models

Abstract

The acquisition of multi-modal biological data for the same sample, such as RNA sequencing and whole slide imaging (WSI), has increased in recent years, enabling studying human biology from multiple angles. However, despite these emerging multi-modal efforts, for the majority of studies only one modality is typically available, mostly due to financial or logistical constraints. Given these difficulties, multi-modal data imputation and multi-modal synthetic data generation are appealing as a solution for the multi-modal data scarcity problem. Currently, most studies focus on generating a single modality (e.g. WSI), without leveraging the information provided by additional data modalities (e.g. gene expression profiles). In this work, we propose an approach to generate WSI tiles by using deep generative models infused with matched gene expression profiles. First, we train a variational autoencoder (VAE) that learns a latent, lower dimensional representation of multi-tissue gene expression profiles. Then, we use this representation to infuse generative adversarial networks (GAN) that generate lung and brain cortex tissue tiles, resulting in a new model that we call RNA-GAN. Tiles generated by RNA-GAN were preferred by expert pathologists in comparison to tiles generated using traditional GANs and in addition, RNA-GAN needs fewer training epochs to generate high-quality tiles. Finally, RNA-GAN was able to generalize to gene expression profiles outside of the training set, showing imputation capabilities. A web-based quiz is available for users to play a game distinguishing real and synthetic tiles: https://rna-gan.stanford.edu/ and the code for RNA-GAN is available here: https://github.com/gevaertlab/RNA-GAN.