henhaobio
Sign in

bioRxivGenomicsPreprint

Systematic comparative analysis of single cell RNA-sequencing methods

Jiarui DingiD, Xian AdiconisiD, Sean SimmonsiD, Monika S. KowalczykiD, Cynthia C. Hession, Nemanja Despot Marjanovic, Travis K. HughesiD, Marc H. WadsworthiD, Tyler BurksiD, Lan NguyễniD, John KwoniD, Boaz BarakiD, William H. GeiD, Amanda J. KedaigleiD, Shaina L. CarrolliD, Shuqiang LiiD, Nir HacoheniD, Orit Rozenblatt–RoseniD, Alex K. ShalekiD, Alexandra–Chloé VillaniiD, Aviv RegeviD, Joshua Z. LeviniD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

A multitude of single-cell RNA sequencing methods have been developed in recent years, with dramatic advances in scale and power, and enabling major discoveries and large scale cell mapping efforts. However, these methods have not been systematically and comprehensively benchmarked. Here, we directly compare seven methods for single cell and/or single nucleus profiling from three types of samples - cell lines, peripheral blood mononuclear cells and brain tissue - generating 36 libraries in six separate experiments in a single center. To analyze these datasets, we developed and applied scumi, a flexible computational pipeline that can be used for any scRNA-seq method. We evaluated the methods for both basic performance and for their ability to recover known biological information in the samples. Our study will help guide experiments with the methods in this study as well as serve as a benchmark for future studies and for computational algorithm development.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (23 May 2019) · CC BY 4.0

Seven single-cell and single-nucleus protocols run side by side on the same cell lines, PBMCs and mouse cortex, analysed through one pipeline: 10x Chromium led the high-throughput methods.

Earlier protocol comparisons were run on different samples, used cultured lines only, or stopped at technical metrics without asking whether a method recovers the cell types in a tissue. Here aliquots of the same sample were processed in parallel in one centre, reanalysed through a single pipeline (scumi) rather than each method's own, and judged on recovery of known blood and cortex cell types, including the first comparison of single-nucleus protocols the authors are aware of.

Key results

  1. ShownSensitivity rankings were broadly consistent across the three sample types: plate-based methods detected the most genes per cell, and 10x Chromium led the high-throughput ones.

    At matched reads per cell in PBMCs, median genes per cell were 2406 and 2632 (Smart-seq2) and 2717 and 2545 (CEL-Seq2); among high-throughput methods 10x Chromium v3 was highest (4494 median UMIs, 1482 genes), with inDrops (256 and 568 genes) and Seq-Well (513 and 372) lowest. In cortex nuclei 10x Chromium v2 led with 5126 and 3127 UMIs, 2462 and 1744 genes. · Similar relative ranking of method sensitivity…

  2. ShownAbility to recover known cell types varied between methods, with 10x Chromium v2 the most reliable and sci-RNA-seq the weakest on tissue.

    In PBMC1, among high-throughput methods, 10x Chromium v2 gave the most cell types and highest mean AUC of marker-signature separation, while Seq-Well and inDrops each missed two types; in PBMC2, 10x v2 and inDrops found all. In cortex every method except sci-RNA-seq found all expected types but pericytes (only DroNc-seq, Cortex1); sci-RNA-seq also missed OPCs and microglia and left 7% and 4% of cells unassigned. · Individual methods vary in their ability to…; A comparison of scRNA-seq methods

  3. ShownPooling each PBMC experiment's libraries and re-clustering recovered every cell type in every library, indicating that misses stemmed at least partly from data quality and depth rather than absent cDNA.

    After combining each PBMC experiment's sampled data with Harmony, all cell types were detected per library, and 20 of 25 reassigned populations were confirmed by the original per-library AUC scoring, the exceptions holding roughly 1-2% of cells. Undetected dendritic cells had been absorbed into monocytes. 10x Chromium v2 agreed best with the joint clustering, sci-RNA-seq worst in cortex. · Pooled data analysis across methods enhances…

  4. ShownMultiplet rates were low except in one inDrops run, but methods differed in the fraction of usable reads and in contamination by reads from other cells.

    Observed species-mixing multiplet rates were below 3.5% for all seven methods except the first inDrops run, and under 1% for plate methods; the true rate is likely about twice the observed one. Reads lacking poly(T) spanned 0.7-2.1% (10x Chromium) to 41.2% (Seq-Well) in PBMCs, exonic fractions 46% (inDrops) to 20% (Seq-Well); in cortex nuclei 10x Chromium v2 had 33% and 29% antisense reads, which were discarded. · Mixture experiments enable detection of…; Read structure and alignment reveal efficiency…; Multiplet Detection

  5. ShownOn the PBMC datasets the unified scumi pipeline reproduced the per-method ranking by median genes per cell obtained with each protocol's own pipeline.

    For the PBMC datasets, median UMIs and genes per cell agreed well between scumi and the method-specific pipelines, the ranking of all methods by median genes per cell was identical, and Pearson correlations of gene expression between the pipelines were very high for every dataset, for UMIs and genes; genes per cell matched more closely than UMIs. The tables and correlation plots are not in the extracted text. · Comparison of scumi with standard computational…

Methods, data and reproducibility

Aliquots of one sample were processed in parallel by each protocol on the same day, with a bulk RNA-seq control and two replicates per sample type, and sequenced together. All FASTQ files were reprocessed with scumi (barcode/UMI extraction, STAR or HISAT2 alignment, featureCounts, UMI collapsing), downsampled to equal reads and cells per method, then clustered and scored against marker signatures by AUC.

  • 1:1 mixture of HEK293 and NIH3T3 cell lines, profiled by all seven methods in two experiments · human and mouse · n = two experiments (Mixture1, Mixture2)
  • Frozen human PBMCs, two experiments; sci-RNA-seq excluded after yielding under 10 genes per cell · human · n = two experiments; ~384 cells (plate) and ~3,000 (high-throughput) targeted
  • Nuclei from whole cortex of 1-month-old C57BL/6 male mice, profiled with Smart-seq2, 10x Chromium v2, DroNc-seq and sci-RNA-seq · mouse · n = two experiments; Smart-seq2 gave 295 and 349 nuclei
  • Matched bulk RNA-seq controls for each sample type, used for mitochondrial fraction, annotation and pseudo-bulk comparison · human and mouse · n = one per experiment
  • All libraries generated in the study · human and mouse · n = 36 libraries, ~92,000 cells, six experiments
Methods & tools
scumi, STAR v2.6.1a, HISAT2 v2.0.5, featureCounts (Subread v1.6.2), RSEM v1.3.0, Seurat v2.3.4, Harmony, Cell Ranger, MASS (R, Huber M-estimator), Smart-seq2 and CEL-Seq2, 10x Chromium v2 and v3, Drop-seq, DroNc-seq, Seq-Well, inDrops, sci-RNA-seq
Reagents & environment
Aligners and reference builds are versioned (STAR v2.6.1a, HISAT2 v2.0.5, Subread v1.6.2, RSEM v1.3.0, Seurat v2.3.4) with Cell Ranger references hg19/mm10 v2.1.0, GRCh38 v1.2.0 and mm10 v1.2.0. Clustering parameters (k, resolution, principal components) are given per sample type.
Reproducibility gaps
No URL is available for the scumi package or the datasets: the text says the data allow open sharing and mentions downloading Cell Ranger references, but no addresses survive in the extracted text. The supplementary tables and figures holding most metrics, costs and times are absent.
Limitations (6)
  • The seven protocols were widely used methods chosen partly for in-house expertise and resources, and the original two-round version of sci-RNA-seq was used, which the authors say may need optimisation for samples such as PBMCs. · stated by the authors
  • Equal cell numbers and human/mouse ratios could not be achieved across methods in the mixture experiments, which affects multiplet rates; the reported rate is observed, not actual. · stated by the authors
  • Plate-based libraries profiled too few cells to reach rare PBMC populations, so their cell-type recovery is not comparable with the high-throughput methods there. · stated by the authors
  • Some comparisons are unbalanced: Seq-Well PBMC1 was sampled at ~46,000 reads per cell versus ~69,000 for other high-throughput methods, Seq-Well PBMC2 and DroNc-seq Cortex2 recovered fewer cells, and 10x v3 was tested only in a limited side-by-side run. · stated by the authors
  • Two replicates per sample type at one centre support rankings but not formal statistical comparison; no confidence intervals or tests are reported in the text. · noted in this summary
  • Cell-type recovery depends on choices the authors tuned per dataset (clustering parameters searched for the best recovery, marker-based identity assignment), so the AUC scores are not fully independent of analyst judgement. · noted in this summary

Read this if you are choosing a single-cell or single-nucleus protocol for blood or brain tissue, or need a controlled multi-method dataset for benchmarking pipelines and integration methods.

Citations
106

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISMereu '20Tian '19Cao '21Simone '24Ziegenhain '16Choudhury '26
68% semantic matchGenomics
Elisabetta Mereu, Atefeh Lafzi, Cátia Moutinho+37 more·Nature Biotechnology·Apr 6, 2020· 499 citations