A Systematic Evaluation of Single-Cell Batch Integration Metrics and sBEE, A New Metric

Abstract

BackgroundSingle-cell RNA sequencing (scRNA-seq) datasets generated across laboratories and experimental conditions often exhibit batch effects that obscure biological variation. Numerous computational methods have been developed to integrate such datasets, making robust benchmarking essential. Evaluation metrics play a central role in assessing integration quality. However, existing metrics each capture only specific aspects of batch mixing and often rely on assumptions that are violated in practice, such as every cell type being present in every batch, balanced batch composition, or simple cluster geometries. Consequently, benchmarking studies frequently report discordant rankings of integration methods, complicating interpretation and method selection. ResultsWe systematically evaluate widely used batch integration metrics using controlled synthetic scenarios that isolate common integration challenges, including imbalanced batch composition, partial cell-type overlap, heterogeneous cluster densities, and varying cluster geometries. Our analysis reveals distinct strengths, limitations, and failure modes, demonstrating that assessments of batch mixing can change substantially across different data characteristics. These observations motivated the development of sBEE (single-cell Batch Effect Evaluator), a metric that assesses batch mixing at the single-cell level using two complementary measures: local neighborhood composition and cross-batch distance relationships. Across diverse simulated scenarios, sBEE produces stable, interpretable evaluations and remains robust to the failure modes identified in existing metrics. We further validate these findings on multiple real-world scRNA-seq datasets. ConclusionsOur work provides a comprehensive characterization of the behavior of widely used batch integration metrics and introduces sBEE, a robust metric that combines complementary neighborhood and distance-based information to provide more reliable assessments of batch mixing. By identifying the conditions under which existing metrics succeed or fail and providing a unified alternative, sBEE enables more consistent benchmarking of single-cell integration methods. Code and datasets are available at https://github.com/tastanlab/sBEE.