Benchmarking of T-Cell Receptor - Epitope Predictors with ePytope-TCR

Abstract

Understanding the recognition of disease-derived epitopes through T-cell receptors (TCRs) has the potential to serve as a stepping stone for the development of efficient immunotherapies and vaccines. While a plethora of sequence-based prediction methods for TCR-epitope binding exists, their available pre-trained models have not been comparatively evaluated on standardized datasets and evaluation settings. Furthermore, technical problems such as non-standardized input and output formats of these prediction tools hinder interoperability and broad usage in applied research. To alleviate these shortcomings, we introduce ePytope-TCR, an extension of the vaccine design and immuno-prediction framework ePytope. We integrated 18 TCR-epitope prediction methods into this common framework offering interoperable interfaces with standard TCR repertoire data formats. We showcase the applicability of ePytope-TCR by evaluating the performance of the prediction methods on two challenging datasets for annotating single-cell repertoires and predicting TCR cross-reactivity towards mutated epitopes. While novel predictors successfully predicted binding to frequently observed epitopes, all methods failed for less observed epitopes. Further, we detected a strong bias in the prediction scores between different epitope classes. We envision this benchmark to guide researchers in their choice of a predictor for a given setting. Further, we aspire to accelerate the development of novel prediction models by allowing fast benchmarking against existing approaches through common interfaces and defining standardized evaluation settings.