scMARK an 'MNIST' like benchmark to evaluate and optimize models for unifying scRNA data

Abstract

Todays single-cell RNA analysis tools provide enormous value in enabling researchers to make sense of large single-cell RNA (scRNA) studies, yet their ability to integrate different studies at scale remains untested. Here we present a novel benchmark dataset (scMARK), that consists of 100,000 cells over 10 studies and can test how well models unify data from different scRNA studies. We also introduce a two-step framework that uses supervised models, to evaluate how well unsupervised models integrate scRNA data from the 10 studies. Using this framework, we show that the Variational Autoencoder, scVI, represents the only tool tested that can integrate scRNA studies at scale. Overall, this work paves the way to creating large scRNA atlases and off-the-shelf analysis tools.