Critical Assessment of ML models for ADMET Prediction in TDC leaderboards

Abstract

In this work we performed a critical assessment of the benchmarking procedures used in Therapeutics Data Commons (TDC) ADMET leaderboards, focusing on reproducibility, robustness against data leakage, and signs of test-set overfitting across all 22 TDC ADMET endpoints. For each endpoint, the top 3 leaderboard models were screened with a unified protocol: execution environment reproducibility check, data leakage assessment, verification of hyperparameter optimisation practices, and final re-evaluation of results and TDC ranking. Only 3 methods (CaliciBoost, MapLight, MapLight+GNN) passed all checks and showed overall reproducible performance, whereas most of top-ranked models exhibited unavailable code, non-reproducible execution environments, runtime incompatibilities, or various methodological flaws. In particular, we identified direct or indirect data leakages in MiniMol, GradientBoost and XGBoost models. We also used our in-house models based on the Mol2Vec architecture to investigate the consequences of deliberately overfitting on the TDC test set. It is shown that deliberate or accidental tuning on the public test set may lead to significant inflation of the model metrics and leaderboard position. Our results emphasize the urgent need for better public ADMET benchmarks with the hidden test sets, strict dataset versioning and model submission with standardized inference environments.