Evaluation design shapes estimates of cross-donor generalization in single-cell classification of cell-type

Abstract

Motivation: Single-cell cell-type classifiers are often evaluated using random cell-level splits, although cells from the same donor are not independent. We asked how evaluation design affects estimates of cross-donor generalization. Results: Across PBMC and lung datasets, random cell-level splitting consistently yielded higher macro F1 than donor-held-out evaluation across multiple expression representations. This gap persisted after batch covariate adjustment and donor-level mixed-effects modeling. In contrast, evaluation gaps were generally minimal in highly separable, trivially easy kidney, pancreas, and blood classification settings and in a randomized-label setting with no meaningful predictive signal. These results show that evaluation bias depends on the underlying classification difficulty and that donor-held-out evaluation is necessary when claims concern generalization to unseen individuals. Availability and Implementation: Code and analyses are available on GitHub at https://github.com/danielchen05/scRNA-cross-donor-generalization/. Processed data are archived at Zenodo at https://doi.org/10.5281/zenodo.19717558. Contact: ajbattle@jhu.edu