Building computational benchmarks: an Omnibenchmark reimplementation of a single-cell preprocessing pipeline evaluation

Background: In the past few years, we have seen a veritable surge in single-cell (e.g., RNA-sequencing) techniques and datasets, enabling increasingly detailed characterization of cellular heterogeneity across tissues and conditions. This surge in single-cell techniques has been complemented by a large number of analysis frameworks and pipelines, and a large parameter space and researcher degrees of freedom to use them. Many neutral benchmarks have been presented for various computational tasks, but most make design decisions that render them incompatible with each other, e.g., through the use of different datasets, metrics, or parameter sets. Results: We showcase Omnibenchmark, a recently developed framework to build reproducible, extensible, and standardized method comparisons for an arms-length benchmark that evaluates five single-cell RNA sequencing pipelines, spanning filtering, normalization, dimensionality reduction, and clustering, on three datasets. The framework facilitates the broad investigation of pipelines used in single-cell data analysis while highlighting how the process of building benchmarks can be streamlined and unified. This standardization enables benchmarks to be extended in several directions, including broader parameter sweeps, comparisons across software versions and architectures, isolation of pipeline steps, and integration of additional pipelines, datasets, and metrics. As an example, our harmonization of pipelines highlighted that tools implement the (clustering) resolution parameter differently. Conclusions: Omnibenchmark provides a reproducible, extensible, and standardized approach for benchmarking. By unifying the process of benchmark construction, it can facilitate more consistent method comparisons and provides a foundation for broader, more flexible, and more sustainable benchmarking efforts.