henhaobio
Sign in

bioRxivBioinformaticsPreprint

Dataset structure outweighs method choice in single-cell cell-type annotation

Oliver WardhanaiD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

Automated cell-type annotation is a prerequisite for nearly all single-cell RNA-sequencing (scRNA-seq) analysis, and the proliferation of annotation tools -- spanning marker-based, similarity-based, classical machine-learning, deep-learning, semi-supervised, large language model (LLM), and transformer foundation-model families -- has exceeded any existing guidance on how to select among them. Existing benchmarks utilize convenient, well-known datasets in which cell number, class imbalance, cell-type number, and differential-expression strength vary concurrently, so performance cannot be attributed to any dataset property. We benchmarked 63 tools from seven families using a Taguchi L9(34) orthogonal array that varied these four properties independently and extended the comparison under more realistic conditions: real data, cross-platform transfer, public marker databases, LLM annotation, and fine-tuned foundation models. With standardized preprocessing and inputs, the leading cell-level families performed within 0.06 {kappa} of one another, and fine-tuned foundation models averaged only slightly higher. Across families, sequencing platforms, and data types, accuracy was predicted near-linearly by how separable cell types were in a shared expression embedding space, measured as k-nearest-neighbor (kNN) purity (R2 = 0.85-0.99). We found that dataset identity accounted for {approx}84% of {kappa} variance while tool identity accounted for {approx}4%. Strong performers were distributed across families, for example, Seurat label transfer and singleCellNet among reference-based classifiers and mLLMCelltype among LLM annotators. When cell types cleanly separate, many annotation methods suffice; when they do not, algorithmic complexity struggles to compensate for noisy signal. Method selection should therefore prioritize available resources, such as curated references and compute availability, over sophisticated modeling.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (2 Oct 2026) · CC BY 4.0

In a 63-tool benchmark, dataset identity explained ≈84% of Cohen's κ variance on synthetic data versus ≈4% for tool identity, and k-nearest-neighbor (kNN) purity predicted accuracy near-linearly (R² = 0.85–0.99).

Prior benchmarks ranked annotation tools inconsistently and used datasets whose cell count, class imbalance, number of cell types and differential-expression strength vary together, so accuracy could not be attributed to tool or data. Here the four properties vary independently in a Taguchi orthogonal design, each tool gets standardized preprocessing and inputs, and findings are rechecked in real, cross-platform, marker-database, large language model (LLM) and foundation-model settings.

Key results

  1. ShownWhich dataset is annotated explains far more of the variation in accuracy than which tool is used. This also held when the reference and query came from different sequencing platforms.

    Crossed dataset × tool random-effects model on synthetic data: dataset ≈84% of κ variance, tool × dataset ≈9%, tool ≈4%, replicate ≈2%. Across 55 configurations trained on one platform and tested on another, median κ was 0.99 (CellBench), 0.90 (pancreas) and 0.62 (PBMCbench). Platform pairs within a block showed no systematic difference. · Cell-type separability dominates the variance of…; Cross-platform annotation transfer is still…; Figure legends

  2. ShownAccuracy rises almost linearly with kNN purity in every family, on both synthetic and real data. kNN purity is the share of each cell's 15 nearest neighbors in a 30-component PCA embedding that carry its label, averaged over all cells, and it holds the largest unique share of κ variance.

    Family-mean κ vs purity per dataset: R² = 0.88–0.98 (synthetic) and 0.85–0.99 (real, n = 9 datasets per fit), Spearman ρ 0.92–1.00, all p ≤ 0.001. Pooled Type III η²: purity 0.71 (synthetic) and 0.39 (real) vs 0.06 and 0.14 for tool identity. Other descriptors had η² ≤ 0.05 within every family. Among design factors, differential-expression difficulty had η² = 0.667, imbalance 0.004. · Cell-type separability dominates the variance of…; Metrics and statistical analysis; Figure legends

  3. ShownThe four leading cell-level tool families differ little in accuracy (marker-based tools trail), and the strongest individual tools come from several families.

    Real-data mean κ: similarity-based 0.676, semi-supervised 0.675, classical machine learning 0.638, deep learning 0.622, marker-based 0.484. The top four are within 0.054 (within 0.026 on synthetic data). Per-dataset means spanned 0.21 to 0.91. Seurat label transfer, singleCellNet, scBalance and scArches ranked near the top on both data types; synthetic vs real tool rank ρ = 0.67. · Annotation accuracy varies more across datasets…; Discussion

  4. ShownFive fully fine-tuned foundation models scored in a narrow band, slightly above every conventional family mean on real data. The best conventional tools still beat their average on the structurally easy datasets.

    κ: scFoundation 0.777, Geneformer V2 0.772, scGPT 0.767, C2S 0.741, scBERT 0.739. Family mean 0.759 vs 0.484–0.676 for the cell-level families. Best single conventional tools reached κ 0.85–0.92 on the easy datasets, above the foundation-model mean there. Mean κ vs kNN purity: ρ = 0.98, R² = 0.98. · Fine-tuned foundation models are the strongest…; Figure legends

  5. ShownWithout a matched reference, LLM annotators given cluster markers beat marker tools driven by a public database. Multi-agent and consensus LLM pipelines scored above a single-call prompt.

    Ontology-matched score (0/0.5/1, mean of three LLM scorers): LLM family 0.611 vs 0.438 for eight CellMarker 2.0-driven marker tools, with LLMs ahead on all nine datasets. mLLMCelltype 0.643, CASSIA 0.613, GPTCelltype 0.578; best database tool scType 0.618. Purity still tracked scores (ρ = 0.77, R² = 0.58 database; ρ = 0.85, R² = 0.60 LLM). · Cell-type separability dictates performance even…; Database-driven marker and LLM annotation; Figure legends

Methods, data and reproducibility

63 tool configurations from seven families ran in one harness with identical preprocessing and best-case inputs (stratified 80/20 reference, top-20 derived markers) on nine Splatter scenarios from a Taguchi L9(3⁴) array, nine profile-matched real datasets, and cross-platform, marker-database, LLM and fine-tuned foundation-model settings. Cohen's κ was decomposed by Type III η² and a random-effects model.

  • Splatter-simulated scRNA-seq in nine Taguchi L9(3⁴) scenarios (500/3,000/15,000 cells; 5/15/30 cell types; imbalance ratio up to 20; DE fold-change 4.8×/2.4×/1.6×) · n = 9 scenarios, 3 seed replicates each
  • Real scRNA-seq validation panel (including Marques) chosen from candidates in the Bioconductor scRNAseq package and elsewhere by Borda-consensus matching to the synthetic scenarios · n = 9 datasets selected from forty-four screened candidates
  • CellBench: lung cancer cell-line mixtures sequenced with 10x v2, CEL-seq2 and Drop-seq (cross-platform block)
  • Pancreas block: Baron (inDrop) paired with Segerstolpe (Smart-seq2)
  • PBMCbench: 10x v2 reference vs Drop-seq, inDrops and Seq-Well on matched PBMC samples
  • CellMarker 2.0 public marker database, tissue-subsetted per scenario
Methods & tools
Seurat, Scanpy, Splatter, CellMarker 2.0, singleCellNet, scArches, scType, mLLMCelltype, CASSIA, scGPT, Geneformer V2, scFoundation
Reagents & environment
R 4.5.1, Seurat 5.0, Bioconductor 3.21; Python tools on AnnData/Scanpy objects (versions not stated); simulation seeds 42, 123, 999.
Assay conditions & compute
R pipelines on an Apple M1 Pro laptop; Python tools on a workstation with 500 GB RAM, a 56-core Intel Xeon E5-2690 v4 and an NVIDIA GeForce RTX 5060; foundation-model fine-tuning on Google Colab NVIDIA L4 (22.5 GB VRAM) and, for C2S, RTX PRO 6000 Blackwell (102 GB VRAM).
Reproducibility gaps
No accessions in the main text (real data via Bioconductor scRNAseq or original publications); Python library versions not stated; Data Availability names a GitHub repo (scRNA-Annotation-Benchmarking-Runbook) but prints the URL with a stray space, so none is listed.
Limitations (6)
  • Each tool received best-case inputs derived from the same original dataset as the test partition (an 80/20 training split sharing gene space, label set and class proportions; top-20 derived markers; ground-truth cluster identities), so reported accuracies are upper bounds rather than expected performance in practice. · stated by the authors
  • Simulations omit batch, donor, doublet and cell-cycle variation, and the L9 array is a main-effects design that assumes negligible interactions among the varied properties. · stated by the authors
  • Synthetic clusters are cleaner and better separated than real ones, which inflates absolute κ in the easy scenarios. kNN purity is a downstream proxy for separability, not an independent cause. · stated by the authors
  • LLM annotators label clusters, so upstream clustering mistakes carry into their output. They are weaker on rare or tissue-restricted populations and fail where labels lack clean Cell Ontology equivalents (e.g. Marques). Their ontology score is not directly comparable with κ. · stated by the authors
  • The real-data family comparisons and purity fits rest on nine datasets selected to match the synthetic scenarios (n = 9 per fit); only the three cross-platform blocks add further real data. · noted in this summary
  • Ontology-matched scores for the LLM and database-driven tools come from three LLM judges (Gemini 3 Flash, Claude Haiku 4.5, GPT-5.4-nano); the main text reports no check against human curators. · noted in this summary

Read this if you pick or build scRNA-seq cell-type annotation tools and want a design-controlled estimate of how much the tool matters relative to the dataset, plus a kNN-purity check for expected accuracy before annotating.

Citations
1

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISHuang '20Abdelaal '19Shen '25Cao '20Ji '23Shasha '21
78% semantic matchBioinformatics
Qianhui Huang, Yu Liu, Yuheng Du+1 more·Genomics Proteomics & Bioinformatics·Dec 24, 2020· 149 citations