The Specification Game: Rethinking the Evaluation of DrugResponse Prediction for Precision Oncology

Abstract

Precision oncology plays a pivotal role in contemporary healthcare, aiming to optimize treatments for each patient based on their unique characteristics. This objective has spurred the emergence of various cancer cell line drug-response datasets, driven by the need to facilitate pre-clinical studies by exploring the impact of multi-omics data on drug response. Despite the proliferation of machine learning models for Drug Response Prediction (DRP), their validation remains critical to reliably assess their usefulness for drug discovery, precision oncology and their actual ability to generalize over the immense space of cancer cells and chemical compounds. This paper shows that the commonly used evaluation strategies for DRP methods learn solutions that optimize an unintended DRP score and fail to predict the proper drug-response activity ("specification gaming"). This problem hinders the advancement of the DRP field, and here we propose a new validation paradigm composed by three Aggregation Strategies (Global, Fixed-Drug, and Fixed-Cell Line) and three train-test Splitting Strategies to ensure a realistic assessment of the prediction performance. We also scrutinize the challenges associated with using IC50 as a prediction label, showing how its close correlation with the drug concentration ranges worsens the risk of misleading performance assessment. We thus propose also an alternative prediction label for DRP which is safer from this perspective.