Single Cell Foundation Models Evaluation (scFME) for In-Silico Perturbation

Abstract

Foundation models pre-trained on large single-cell RNA atlases offer a compelling alternative to in-vitro experimentation for understanding gene regulatory networks and conducting gene perturbation analyses, with significant implications for target identification. Numerous foundation models have been developed, building upon early efforts such as Geneformer and scGPT. Hyperparameter optimization also results in multiple variants which require comparative analysis. Current benchmarking approaches focus on feature-based assessments or intuitive biological and statistical tasks, which may not align with the models training objectives. A recent study proposed a systematic benchmarking framework; however, its scope was limited to pre-trained (zero-shot) models. To address these limitations, we propose Single-Cell Foundation Model Evaluation (scFME)--a systematic method designed to benchmark fine-tuned foundation models for insilico perturbation (ISP). scFME ensures comprehensive and robust assessment by requiring sufficient separation between control and perturbed cells at the outset and by quantifying ISP accuracy against zero and random perturbation baselines. Furthermore, scFME enables exploration of model performance across distinct gene categories, facilitating biological interpretation and functional relevance. Using this framework, we evaluated several commonly used models (and some of their variants) and demonstrated that the methodology effectively characterizes their performance in ISP studies. Our results position scFME as a versatile and rigorous methodology for evaluating and comparing current and future foundation models.