Multifaceted Analysis of Training and Testing Convolutional Neural Networks for Protein Secondary Structure Prediction

Abstract

Protein secondary structure prediction remains a vital topic with improving accuracy and broad applications. By using deep learning algorithms, prediction methods not relying on structure templates were recently reported to reach as high as 87% accuracy on 3 labels (helix, sheet or coil). Due to lack of a widely accepted standard in secondary structure predictor development and evaluation, a fair comparison of predictors is challenging. A detailed examination of factors that contribute to higher accuracy is also lacking. In this paper, we present: (1) a new test set, Test2018, consisting of proteins from structures released in 2018 with less than 25% similar to any protein published before 2018; (2) a 4-layer convolutional neural network, SecNet, with an input window of {+/-}14 amino acids which was trained on proteins less than 25% identical to proteins in Test2018 and the commonly used CB513 test set; (3) a detailed ablation study where we reverse one algorithmic choice at a time in SecNet and evaluate the effect on the prediction accuracy; (4) new 4- and 5-label prediction alphabets that may be more practical for tertiary structure prediction methods. The 3-label accuracy of the leading predictors on both Test2018 and CB513 is 81-82%, while SecNets accuracy is 84% for both sets. The ablation study of different factors (evolutionary information, neural network architecture, and training hyper-parameters) suggests the best accuracy results are achieved with good choices for each of them while the neural network architecture is not as critical as long as it is not too simple. Protocols for generating and using unbiased test, validation, and training sets are provided. Our data sets, including input features and assigned labels, and SecNet software including third-party dependencies and databases, are downloadable from dunbrack.fccc.edu/ss and github.com/sh-maxim/ss.