Do Geometric Outliers Identify Important Genes in Single-Cell Foundation Models?

Abstract

Single-cell foundation models (scFMs) produce gene embeddings that are increasingly interpreted biologically, but it is unclear whether different models organise gene space consistently, or whether geometric extremes identify important genes. The same four-metric screen was applied to Geneformer, scGPT and scFoundation, which differ in architecture, training objective and expression encoding. The models agreed weakly on individual outlier genes but showed structured class-level convergence: ribosomal enrichment occurred in all three, although weaker and caller-dependent in scFoundation, while strong mitochondrial enrichment was confined to Geneformer and scGPT. Gene-level agreement was itself uneven, with Geneformer and scGPT overlapping above chance and scGPT and scFoundation not, and caller stability differed between models. Most outliers were not extreme in ESM-2 protein-sequence space, indicating principally model-specific rather than protein-sequence geometry. These patterns did not translate into the tested notions of gene importance: deleting the highest-anomaly Geneformer genes did not impair cell-type annotation beyond matched controls, and covariate-adjusted outlier status was not associated with ClinVar membership in any model. Gene-embedding geometry therefore characterises model-specific structure, but does not provide standalone evidence of downstream leverage or disease relevance.