Predicting Pre-treatment Resistance or Post-treatment Effect? A Systematic Benchmarking of Single-Cell Drug Response Models

Intratumoral heterogeneity drives variable drug responses in cancer. Single-cell RNA sequencing (scRNA-seq) enables characterization of such heterogeneity and prediction of drug response at single-cell resolution. Accordingly, various computational models have been developed to infer drug response from scRNA-seq data. However, their performance, robustness, and generalizability across different biological contexts remain insufficiently evaluated. To address this gap, we benchmarked representative single-cell drug response prediction models using 26 curated datasets comprising over 760,000 cells across 12 cancer types and 21 therapeutic agents. We constructed balanced and imbalanced scenarios to explore model behavior under different label distributions. To address the lack of ground-truth labels in conventional scRNA-seq datasets, we incorporated two lineage-tracing datasets with experimentally validated drug-response annotations, enabling evaluation in a clinically relevant pre-treatment prediction setting. Our results show that prediction performance was markedly higher in cell lines than in tissue samples and varied substantially across datasets, treatment types, and imbalance levels. Under increasing class imbalance, several methods showed reduced performance, and no single method consistently dominated across all evaluation settings. In lineage-tracing datasets, treatment-associated states were more readily distinguished than lineage-defined pre-sensitive and pre-resistant states, while transcriptomic analyses indicated that pre-existing resistance-related signals were detectable but relatively subtle. For scDEAL, which showed competitive performance across several benchmark settings, label-substitution and ablation analyses further indicated that its performance was jointly influenced by training-label definition and architectural design. Application to an independent PDAC organoid dataset showed that scDEAL predictions were associated with treatment-related transcriptional state changes and biological programs. In summary, our study systematically characterizes the performance and limitations of current single-cell drug response prediction approaches across diverse evaluation settings. These results highlight the influence of dataset characteristics, class imbalance, training design, and the strength of resistance-associated signals on model performance, providing a reference for the evaluation and further development of single-cell drug response prediction methods.