AUC-PR is a More Informative Metric for Assessing the Biological Relevance of In Silico Cellular Perturbation Prediction Models

Abstract

In silico perturbation models, computational methods which can predict cellular responses to perturbations, present an opportunity to reduce the need for costly and time-intensive in vitro experiments. Many recently proposed models predict high-dimensional cellular responses, such as gene or protein expression to perturbations such as gene knockout or drugs. However, evaluating in silico performance has largely relied on metrics such as R2, which assess overall prediction accuracy but fail to capture biologically significant outcomes like the identification of differentially expressed genes. In this study, we present a novel evaluation framework that introduces the AUC-PR metric to assess the precision and recall of DE gene predictions. By applying this framework to both single-cell and pseudo-bulked datasets, we systematically benchmark simple and advanced computational models. Our results highlight a significant discrepancy between R2 and AUC-PR, with models achieving high R2 values but struggling to identify Differentially expressed genes accurately, as reflected in their low AUC-PR values. This finding underscores the limitations of traditional evaluation metrics and the importance of biologically relevant assessments. Our framework provides a more comprehensive understanding of model capabilities, advancing the application of computational approaches in cellular perturbation research.