henhao
Sign in
BioinformaticsbioRxiv
Preprint

Accessible and reproducible deployment reveals the practical boundaries of single-cell foundation models

Siyu HouiD, Penghui YangiD, Wenjing MaiD, Jinxi XiangiD, Jade Xiaoqing WangiD, Hui Wan, Ying MaiD, Xiang ZhouiD

Abstract

Single-cell foundation models (scFMs) have been widely promoted as a unifying paradigm for transcriptomic analysis, yet whether large-scale pretraining translates into reproducible biological advantages remains unclear. Their adoption is further hindered by heterogeneous implementations, preprocessing requirements, and computational environments. Here we develop a unified, automated, and reproducible framework for standardized deployment and controlled evaluation of scFMs across datasets, computational environments, training regimes, and downstream analyses, substantially lowering the technical barriers to their use. Leveraging this framework, we systematically investigate thirteen scFMs alongside established methods across nearly one hundred datasets spanning diverse biological contexts. Our analyses reveal clear practical boundaries to scFM utility. First, increased model scale, architectural complexity, pretraining corpus size, or input encoding does not consistently translate into superior downstream performance. Instead, measurable properties of embedding geometry provide a model-agnostic, representation-level explanation for differences in zero-shot performance across diverse model families. Second, the benefits of pretrained representations depend strongly on the biological and supervision regime: scFMs provide their clearest advantages under extremely limited supervision, particularly for rare-cell annotation and open-set detection of source-absent cell states, whereas established methods remain competitive or preferable in most other settings. Task-matched analyses further show that scFM representations transfer inconsistently to spatial-domain recovery, while their gene embeddings capture broad functional relatedness without reliably recovering context-specific regulatory relationships. Together, these results establish that scFM utility is neither universal nor determined simply by model scale alone, but varies with learned representation geometry, biological context, and supervision. By combining reproducible deployment with large-scale empirical and mechanistic investigation, our framework provides a principled foundation for determining when foundation-model pretraining offers genuine practical value and when simpler approaches remain sufficient.

PDFDOICodeData
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
1
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (2 Sept 2026) · CC BY 4.0

Across 13 single-cell foundation models and 94 datasets, PCA and established methods usually match or beat them; pretrained embeddings help mainly with scarce labels, rare cells and open-set detection.

Single-cell foundation models (scFMs) are hard to install and run consistently, and earlier comparisons, as the authors describe them, covered only a few models such as scGPT, Geneformer and scFoundation on limited tasks and ranked rather than explained performance. The authors package 13 models behind one containerized interface, test them under matched conditions on 94 datasets, and link zero-shot quality to measurable embedding geometry rather than to model scale.

Key results

  1. ShownOn 26 Tabula Sapiens v2 tissues not documented in any model's pretraining corpus, only SCimilarity preserved cell-type structure significantly better than PCA; most frozen scFMs matched or fell below PCA.

    Mean biological-conservation family rank (20 representations, 548,977 cells): SCimilarity 3.98, scFoundation 6.15, PCA 6.60. SCimilarity vs PCA Holm-adjusted P = 0.0145; scFoundation vs PCA adjusted P = 0.816. Geneformer, scPRINT, CellFM, CELLama, GenePT-w and scBERT ranked well below PCA. No frozen scFM improved biological conservation and batch mixing together. · 2.2 A controlled tissue atlas reveals selective…

  2. SuggestedZero-shot quality tracked embedding geometry (directional alignment specific to cell type, and preservation of local expression neighbourhoods). Parameter count, pretraining corpus size, model family and input encoding did not predict it.

    Across 20 representations, cell-type anisotropy contrast ρ = 0.687 (P = 8.15×10^−4) and local neighbourhood preservation RNX ρ = 0.798 (P = 2.44×10^−5; 0.901 among 13 frozen scFMs); RNX also tracked trajectory recovery (ρ = 0.775, nine systems). Corpus size ρ = −0.237 (P = 0.480, n = 11), parameter count ρ = 0.392 (P = 0.211, n = 12); no model-characteristic test survived Holm correction. · 2.3 Embedding geometry, but not model…; 4.5 Embedding-geometry analysis

  3. ShownWith abundant labels, established annotators led and fine-tuning added little. In closed-set transfer, the main gain from scFMs was label efficiency, including better recall of rare cell types with one label per class.

    Data-rich closed-set transfer: CellTypist median macro-F1 0.749 vs 0.720 for scFoundation (best scFM) and 0.704 for PCA. scFoundation, scCello and UCE needed only 5 labels per class to attain 90% of their N = 500 macro-F1 (PCA needed 50); at 1 label per class, rare-type recall was 0.68 (UCE), 0.65 (scFoundation), 0.61 (PCA), 0.48 (CellTypist). Fine-tuning: median macro-F1 change +0.03 across 13 scFMs. · 2.4 Pretrained representations show selective…; 4.4.1 Cross-domain supervised transfer and…

  4. ShownFeature-space distance in the leading frozen scFM embeddings (and pretrained scVI) flagged query cell types missing from the reference better than classifier scores or PCA did. Novel cells that went undetected were mostly given labels from related lineages.

    Across 37 eligible transfer directions, Mahalanobis distance in frozen embeddings gave mean AUROC 0.82–0.84 for scFoundation, LangCell and UCE vs 0.73 for PCA and 0.82 for pretrained scVI; native CellTypist and SingleR scores reached 0.72–0.76 (CellTypist 0.73). Undetected novel cells forced into a reference label landed in the true type's broad class 80% of the time and its coarse compartment over 90% of the time. · 2.4 Pretrained representations show selective…; 4.4.1 Cross-domain supervised transfer and…

  5. ShownGains did not carry over to spatial-domain recovery or gene-level regulatory inference. A spatially native model outperformed every frozen scFM, and no scFM gene output beat GENIE3 on recovering BEELINE regulatory edges.

    Spatial domains, 25 sections: NOVAE mean rank 2.87 vs 5.98 for scFoundation and 7.61 for PCA, itself ahead of 10 of 13 frozen scFMs (NOVAE vs all, Holm-adjusted P = 1.96×10^−4). BEELINE AUPRC ratio: GENIE3 1.27, contextual UCE 1.15, static GenePT-w 1.12. GenePT-w had STRING AUROC 0.827 (co-expression 0.585) but AUROC 0.512 on 570 CRISPR transcription-factor perturbations. · 2.5 scFMs transfer unreliably to spatial…; 2.6 Gene embeddings capture broad functional…

Methods, data and reproducibility

Thirteen scFMs were wrapped behind one AnnData input/output contract, each in a version-pinned Docker or Singularity container run by Nextflow. They were compared with PCA, scVI, Harmony, Seurat, CellTypist, SingleR, NOVAE and GENIE3 across zero-shot clustering, trajectories, cell-type transfer, open-set detection, spatial domains and gene-level tasks, and embedding geometry was correlated with performance.

  • Tabula Sapiens v2 tissue datasets from CZ CELLxGENE (donors shared with v1 removed), used for the zero-shot benchmark · human (inferred from the Tabula Sapiens atlas; not stated in the text) · n = 26 tissues; 548,977 cells · CELLxGENE collection e5f58829-1a66-40b5-a624-9046778e74f5
  • Tabula Sapiens v1 tissue datasets, paired with v2 for 44 directed cross-dataset transfers · human (inferred from the Tabula Sapiens atlas; not stated in the text) · n = 24 tissues · CELLxGENE collection e5f58829-1a66-40b5-a624-9046778e74f5
  • Dynverse developmental trajectory datasets (linear, bifurcating, tree-structured) · n = 9 datasets · Zenodo record 1443566
  • SpatialLIBD dorsolateral prefrontal cortex sections (Visium) · n = 12 sections · https://research.libd.org/spatialLIBD/
  • HER2ST breast cancer sections (ST platform) · n = 8 sections
  • MERFISH hypothalamic preoptic-region sections · n = 5 sections · doi:10.5061/dryad.8t8s248
  • BEELINE experimental datasets (hESC, hHep, mDC, mESC, mHSC-E, mHSC-GM, mHSC-L) for regulatory-edge recovery · n = 7 datasets from five studies · doi:10.5281/zenodo.3378975
  • Norman CRISPRa and Replogle K562 and RPE1 CRISPRi perturbation data (scPerturb release) · not stated in the text (K562 and RPE1 are human cell lines) · n = 3 datasets; 570 single-transcription-factor perturbations · Zenodo record 13350497
Methods & tools
Nextflow, Docker / Singularity, 13 scFMs: scGPT, Geneformer, scFoundation, CELLama, SCimilarity, scBERT, CellFM, LangCell, scCello, scPRINT, CellPLM, GenePT-w, UCE, scVI / scANVI, Harmony, Seurat (CCA, RPCA), CellTypist, SingleR, NOVAE, GENIE3, GRNBoost2, PIDC, Slingshot, PAGA, diffusion pseudotime, Monocle 3 (Dynverse evaluation), Scanpy, PyTorch, HuggingFace Transformers
Code
https://github.com/Svvord/scFoundry
Data & accessions
https://cellxgene.cziscience.com/collections/e5f58829-1a66-40b5-a624-9046778e74f5https://zenodo.org/records/1443566https://github.com/almaan/her2sthttps://datadryad.org/dataset/doi:10.5061/dryad.8t8s248https://doi.org/10.5281/zenodo.3378975https://zenodo.org/records/13350497
Software & environment
Per-model Docker or Singularity containers with fixed library versions; Nextflow records software versions and parameters; hyperparameters kept in configuration files; model versions stated (e.g. scGPT v0.2.4, Geneformer v0.1.0, SCimilarity v0.4.1, scPRINT v2.3.5). Code under MIT license.
Assay conditions & compute
NVIDIA H200 GPUs (140 GB) via Slurm with 256 GB system memory and four CPU cores per job; embedding pipelines also validated on H100 and RTX 5000 Ada GPUs (at least 32 GB). At batch size 32 or lower, all GPU-enabled methods fit on 40 GB GPUs.
Reproducibility gaps
Model-specific optimization settings, open-set detector definitions, full resource profiles, dataset eligibility criteria and annotation-resource versions are in Supplementary Information, not part of this text; no registry for the container images is named.
Limitations (6)
  • Links between geometry and performance are model-level Spearman correlations with nominal, unadjusted P values, read as diagnostics rather than proof that changing a geometric statistic would improve an embedding. · stated by the authors
  • Models differ at the same time in input encoding, training objective and backbone, so the model taxonomy is descriptive and cannot isolate the effect of any single design factor. · stated by the authors
  • Gene-level endpoints test retrieval of recorded regulons and strongly responding genes. They do not test identification of direct targets or prediction of whole post-perturbation transcriptomes. · stated by the authors
  • GenePT-w's strong functional signal may partly reflect knowledge already present in the gene-description text rather than inference from expression data. · stated by the authors
  • The conclusion that scale and corpus size do not predict quality rests on only 11 to 12 models. Non-significant correlations such as parameter count ρ = 0.392 (P = 0.211) have limited power to rule out an effect. · noted in this summary
  • Several models ran only after the authors' own fixes or completed training routes (CellFM limited to its encoder; supervised pipelines completed for scFoundation and CellFM). Rankings therefore reflect these wrappers, not necessarily each model's best use. · noted in this summary

Who should read:Read this if you are deciding whether a single-cell foundation model beats PCA, scVI or CellTypist for your annotation, integration, spatial or GRN analysis, or want containerized wrappers that run 13 scFMs with one command.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISLiu '26Liu '26Shakeel '26Kim '26Gaballa '26Hao '24
84% semantic matchBioinformatics
Tianyuan Liu, Tingze Feng, Xianrun Pan+5 more·bioRxiv·Jul 13, 2026