henhaobio
Sign in

Nature MethodsBioinformaticsPeer-reviewed journal article

Benchmarking atlas-level data integration in single-cell genomics

Malte D. LueckeniD, Maren BüttneriD, Kridsadakorn ChaichoompuiD, Anna DaneseiD, Marta InterlandiiD, M. F. Mueller, Daniel C. StrobliD, Luke ZappiaiD, Martin DugasiD, Maria Colomé‐TatchéiD, Fabian Joachim TheisiD

PDFDOIPreprintCodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Cell atlases often include samples that span locations, labs, and conditions, leading to complex, nested batch effects in data. Thus, joint analysis of atlas datasets requires reliable data integration. Choosing a data integration method is a challenge due to the difficulty of defining integration success. Here, we benchmark 38 method and preprocessing combinations on 77 batches of gene expression, chromatin accessibility, and simulation data from 23 publications, altogether representing >1.2 million cells distributed in nine atlas-level integration tasks. Our integration tasks span several common sources of variation such as individuals, species, and experimental labs. We evaluate methods according to scalability, usability, and their ability to remove batch effects while retaining biological variation. Using 14 evaluation metrics, we find that highly variable gene selection improves the performance of data integration methods, whereas scaling pushes methods to prioritize batch removal over conservation of biological variation. Overall, BBKNN, Scanorama, and scVI perform well, particularly on complex integration tasks; Seurat v3 performs well on simpler tasks with distinct biological signals; and methods that prioritize batch removal perform best for ATAC-seq data integration. Our freely available reproducible python module can be used to identify optimal data integration methods for new data, benchmark new methods, and improve method development.

In-depth summary

Claude Opus 5.5 · max effort · 8 Nov 2026 · from the full text of version 2 (27 May 2020) · CC BY 4.0

BBKNN, Scanorama, and scVI lead complex scRNA-seq integration in a 38-setup, nine-task, 14-metric benchmark; highly variable gene selection helps, whereas scaling favors batch removal over bio-conservation.

Atlas projects pool samples across labs, protocols, donors, and species, producing nested nonlinear batch effects, and with 31 scRNA-seq integration methods available, choosing one was hard and subjective. Earlier benchmarks used only simple batch-removal tasks, where ComBat or Harmony beat nonlinear methods; this work tests methods on complex atlas-scale scRNA-seq and scATAC-seq tasks, adding label-free metrics such as trajectory and cell cycle conservation.

Key results

  1. ShownOn real RNA tasks, graph-based BBKNN and the embeddings from Scanorama and scVI ranked best and LIGER performed poorly, while Seurat v3 and Harmony did well mainly on simulations with distinct batch and biological structure.

    Out of 38 method-preprocessing setups per task (342 attempted runs), each method was ranked by its top preprocessing variant, with overall scores weighting batch removal to bio-conservation 40:60; Seurat v3 and Harmony usually fell outside the top third on real data; rankings under z-score versus min-max metric scaling correlated at Spearman's R >0.94 for all tasks. · BBKNN, Scanorama, and scVI perform best overall…; Single-cell integration benchmarking (scIB); Benchmarking setup; Discussion

  2. ShownRestricting input to highly variable genes (HVGs) generally improved integration, whereas per-batch scaling pushed methods toward batch removal at the cost of biological conservation; full gene sets better retained trajectory and cell cycle signals.

    Comparing runs differing in one preprocessing step across RNA and simulation tasks, HVG runs scored higher in 72% of comparisons overall, 80% for batch removal, and 60% for bio-conservation; scaling raised batch removal in 63% of comparisons but lowered bio-conservation in 72%. · Scaling improves batch removal but impairs…; Discussion

  3. ShownBatch removal and bio-conservation traded off: BBKNN and Seurat v3 favored mixing, Scanorama and Conos favored bio-conservation; the strongest effects (species, location, nuclei vs cells, microwell-seq) were mainly merged by LIGER and Seurat v3 at the cost of biological signal.

    In the lung task Seurat v3 merged airway and parenchymal endothelial cells whereas Scanorama kept their spatial differences; in the human/mouse immune task only LIGER and Conos integrated species, LIGER merging neutrophils with monocytes and losing the trajectory; Scanorama and scVI integrated mouse brain nuclei and cells while retaining rare cell types. · The trade-off between batch removal and…; Discussion

  4. ShownNo method integrated scATAC-seq data adequately; BBKNN, Harmony, and scVI removed batch effects best, while bio-conservation scores were often driven by compact but unintegrated clusters, leading the authors to prioritize batch correction for ATAC.

    Across two mouse brain tasks (25,960 cells at a 13:57:30 dataset ratio; 67,612 cells at 5:20:75), trVAE and LIGER failed on the large task and MNN on both; Seurat v3 and ComBat led bio-conservation but ranked fourth and fifth for batch correction, and most batches stayed separated in UMAPs. · scATAC-seq batch effects require strong batch…; Mouse brain integration task (ATAC); Discussion

  5. ShownMethods differed sharply in scalability and usability: only BBKNN completed every dataset and preprocessing combination, trVAE and Seurat v3 hit dataset-size ceilings, and Harmony, BBKNN, and Seurat v3 scored best for usability.

    With up to 323 GB memory and 96 hours per run, trVAE could not integrate over 67,000 cells and Seurat v3 failed above 100,000; Conos used the most memory yet integrated 1 million unscaled cells; on unscaled data scVI had the best memory use and BBKNN and ComBat the fastest runtimes. · Scalability and usability; Benchmarking setup; Usability assessment

Methods, data and reproducibility

Ten integration methods, each with up to four preprocessing variants (HVG selection, per-batch scaling), were run on nine tasks (five RNA-seq, two scATAC-seq, two simulated); graph, embedding, and corrected-matrix outputs were scored with 14 batch-removal and bio-conservation metrics, including graph-adapted kBET and LISI, weighted 40:60.

  • Six human pancreas scRNA-seq datasets (Satija and Hemberg lab collections); each dataset, or each inDrop donor, treated as a batch · human · n = 16,382 cells · GSE81076, GSE85241, GSE86469, E-MTAB-5061, GSE84133, GSE81608
  • Human immune cells from bone marrow and peripheral blood across five datasets (e.g., Oetjen, Villani, 10X Genomics), with donors as batches · human · n = 33,506 cells; 10 batches
  • Human/mouse immune task adding Dahlin and Mouse Cell Atlas samples (eight datasets in total), keeping 8,135 genes mapped between species via biomaRt · human and mouse · n = 97,952 cells
  • Lung atlas (Vieira Braga et al.): 10X and Drop-seq parenchymal transplant samples plus 10X airway biopsies from three labs; donors as batches · human (donors, per Table 1 legend) · n = 32,472 cells from 16 donors · GSE130148 (Drop-seq data only)
  • Mouse brain scRNA-seq and snRNA-seq from four studies (Rosenberg SPLiT-seq, Zeisel, Saunders Drop-seq, Tabula Muris) spanning 15 annotated regions · mouse · n = 978,734 cells · GSE110823 (Rosenberg et al.)
  • Small mouse brain scATAC-seq task: six samples from Cusanovich (combinatorial indexing), 10X Genomics, and Fang (single-nucleus) datasets · mouse · n = 25,960 cells; 57,070 features · GSE111586 (Cusanovich et al.)
  • Large mouse brain scATAC-seq task: all 11 samples from the same three datasets, with one dataset contributing most cells · mouse · n = 67,612 cells; 57,447 features · GSE111586 (Cusanovich et al.)
  • Splatter (Splat) simulations: one task with uneven cell group proportions and library sizes across batches, and one nested sub-batch task · n = 12,097 cells, six batches; 19,318 cells, 16 sub-batches
Methods & tools
scIB Python module and Snakemake benchmarking pipeline, kBET 0.99.6 (k-nearest-neighbor batch effect test) and graph LISI (local inverse Simpson's index), BBKNN 1.3.5 (batch-balanced k-nearest neighbors), Scanorama 1.4, scVI 0.5.0, Seurat v3 (3.1.1), Harmony 1.0, Conos 1.2.1 (clustering on network of samples), LIGER 0.4.2, trVAE 0.0.1 (transformer variational autoencoder), MNN (mutual nearest neighbors; mnnpy 0.1.9.5) and ComBat (Scanpy 1.4.5), Splatter, scran 1.10.2, Scanpy, EpiScanpy 0.1.10 (simulation and preprocessing)
Software & environment
Scanpy 1.4.4/1.4.5, scran 1.10.2, biomaRt 2.38.0, EpiScanpy 0.1.10, scikit-learn 0.22.1, scipy 1.3.0, kBET 0.99.6; R and Python methods run in separate conda environments, bridged by anndata2ri, within a Snakemake workflow.
Assay conditions & compute
Each run had 12 cores/24 threads, 16 GB memory per core plus 131 GB shared swap (up to 323 GB total) and a four-day (96-hour) limit; over-limit runs were scored NA; trVAE did not scale to large tasks without GPU hardware.
Reproducibility gaps
Lung 10X counts (SoupX-corrected) came directly from the original authors; immune dataset retrieval details sit only in Supplementary Data 4; several sources (e.g., Tabula Muris on figshare) are cited by site and download date, not accession.
Limitations (6)
  • Graph outputs from BBKNN and Conos support only a subset of metrics, so their scores may be less robust, and silhouette-based bio-conservation metrics reward compact but unintegrated clusters when integration fails, as in the ATAC tasks. · stated by the authors
  • Without GPU hardware, trVAE could not be benchmarked on the larger, complex tasks, and deep learning methods were not tuned per method (to avoid bias), so their potential at million-cell scale may be understated. · stated by the authors
  • ATAC integration relied on highly variable windows shared across datasets, which become fewer as batches and cells increase; methods designed for graded expression counts received binary-derived input, and gene-level ATAC features were left untested. · stated by the authors
  • Whether species, spatial location, or tissue counts as batch or biology is subjective, so the fixed definitions used here may not suit analyses where retaining such variation is preferable. · stated by the authors
  • Ranking each method by its single best preprocessing setup lowers Seurat v3 and Harmony, for which no consistently optimal preprocessing scheme was found across tasks. · stated by the authors
  • The 40:60 batch-removal to bio-conservation weighting is a design choice; only z-score versus min-max metric scaling was checked, so how method rankings shift under other weightings is not reported. · noted in this summary

Read this if you need to choose or develop a single-cell integration method for multi-lab atlas data, or want reusable metrics (graph kBET/LISI, trajectory and cell cycle conservation) for benchmarking batch correction.

Citations
1,441
FWCI
55.86
Field percentile
top 1%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISPolański '19Specque '25Li '22Xiao '24Myradov '26Lee '23
70% semantic matchBioinformatics
Krzysztof Polański, Matthew D. Young, Zhichao Miao+3 more·Bioinformatics·Aug 8, 2019· 933 citations