A study on experimental bias in post-translational modification predictors

Protein post-translational modifications (PTMs) play an important role in numerous biological processes by significantly affecting protein structure and dynamics. Effective computational methods that provide a sequence-based prediction of PTM sites are desirable to guide functional experiments. Whereas these methods typically train neural networks on one-hot encoded amino acid sequences, protein language models carry higher-level pattern information that may improve sequence based prediction performance and hence constitute the current edge of the field. In this study, we first evaluate the training of convolutional neural networks on top of various protein language models for sequence based PTM prediction. Our results show substantial prediction accuracy improvements for various PTMs with current procedures of dataset compilation and model performance evaluation. We then used model interpretation methods to study what these advanced models actually base their learning on. Importantly for the entire field of PTM site predictors trained on proteomics-derived data, our model interpretation and transferability experiments reveal that the current approach to compile training datasets based on proteomics data leads to an artefactual protease-specific training bias that is exploited by the prediction models. This results in an overly optimistic estimation of prediction accuracy, an important caveat in the application of advanced machine learning approaches to PTM prediction based on proteomics data. We suggest a partial solution to reduce this data bias by implementing negative sample filtering, only allowing candidate PTM sites in matched peptides that are present in the experimental metadata. Availability and implementationThe prediction tool, with training and evaluation code, trained models, datasets, and predictions for various PTMs are available at https://github.com/jasperzuallaert/PhosphoLingo. Contactsven.degroeve@vib-ugent.be and nico.callewaert@vib-ugent.be Supplementary informationSupplementary materials are available at bioRxiv.