henhaobio
Sign in

bioRxivBioinformaticsPreprint

Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks

Ananthan NambiariD, Simon LiuiD, Mark Hopkins, Maeve Heflin, Sergei MasloviD, Anna RitziD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

AO_SCPLOWBSTRACTC_SCPLOWThe scientific community is rapidly generating protein sequence information, but only a fraction of these proteins can be experimentally characterized. While promising deep learning approaches for protein prediction tasks have emerged, they have computational limitations or are designed to solve a specific task. We present a Transformer neural network that pre-trains task-agnostic sequence representations. This model is fine-tuned to solve two different protein prediction tasks: protein family classification and protein interaction prediction. Our method is comparable to existing state-of-the art approaches for protein family classification, while being much more general than other architectures. Further, our method outperforms all other approaches for protein interaction prediction. These results offer a promising framework for fine-tuning the pre-trained sequence representations for other protein prediction tasks.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (16 Jun 2020) · CC BY 4.0

A RoBERTa-style Transformer, pre-trained on 450K Swiss-Prot sequences in 18 hours on 4 V100 GPUs, rivals DeepFam at family classification and beats PIPR at human protein-protein interaction prediction.

Under 1% of UniProtKB sequences are experimentally characterized, and prior deep learning models for protein family classification and protein-protein interaction (PPI) prediction are, per the authors, task-specific or costly to pre-train. PRoBERTa instead pre-trains one task-agnostic representation (byte pair encoding plus RoBERTa-style masked language modeling) and fine-tunes it for both tasks, at a reported 170-fold pre-training efficiency gain over a prior BERT-based protein model (Rives, 2019).

Key results

  1. ShownOn human HIPPIE interactions with random human protein pairs as negatives (conservative scenario), fine-tuned PRoBERTa predicts PPIs from sequence pairs better than PIPR and two ProtVec/DeepFam-based baselines.

    Trained on 20% of the training/validation data (108,103 examples, 18% of the total): accuracy 0.96, precision 0.93, recall 0.94, ROC AUC 0.99; the authors report a large margin over PIPR, ProtVec+DeepInteract and DeepFam-transfer+DeepInteract (competitor values are in Figure 5, not in the text). With the full training set, accuracy 0.99. · 3.3.1 Conservative Scenario; 3.4 PRoBERTa Scalability and Robustness; Tables

  2. ShownWith harder negatives (unconnected HIPPIE protein pairs, aggressive scenario), performance drops, but PRoBERTa still ranks above PIPR, which in turn beats the two other baselines.

    20% training data: accuracy 0.79, precision 0.82, recall 0.73, ROC AUC 0.87; the full training set raises accuracy to 0.84. Competitor values are in Figure 6, not in the text. · 3.3.2 Aggressive Scenario; 3.4 PRoBERTa Scalability and Robustness; Tables

  3. SuggestedThe high conservative-scenario scores may partly reflect learning which proteins occur in HIPPIE rather than learning interactions themselves.

    The conservative-trained model scored accuracy 0.59, precision 0.54 and recall 0.94 on the aggressive dataset; together with a larger NMI drop against protein families after aggressive fine-tuning (Figure 4), the authors read this as the conservative model doing something close to protein classification. · 4 Discussion; 3.3.1 Conservative Scenario; 3.3.2 Aggressive Scenario

  4. ShownFine-tuned for multi-class protein family classification, PRoBERTa is comparable to the CNN-based DeepFam and better than a simple CNN; pre-trained embeddings fed to per-family logistic regression beat ProtVec and match ProtDocVec and ProtFreqVec.

    Multi-class: accuracy 0.92 on the 10% test split of 313,214 single-family Swiss-Prot proteins (DeepFam and four-layer CNN values are in Table 2, not in the text). Binary (families with >50 proteins, 30% held out): weighted mean accuracy 0.98, lowest family accuracy 0.77 (57 proteins); better than ProtVec, similar to ProtDocVec and ProtFreqVec. · 3.2 Protein Family Classification; 2.4.1 Task: Protein Family Classification; Fine-tuning Data; Tables

  5. ShownBefore any fine-tuning, the masked-language-model embeddings already group proteins by family; family fine-tuning sharpens this grouping, while PPI fine-tuning weakens it.

    k-means (k = 4000) on 20-dimensional PCA of embeddings for 9,151 human UniProt proteins (3,860 annotated families), using a d = 192 model variant: mean NMI 0.798213 over 20 runs, above random cluster assignment (t-test p = 8 × 10^−180). Fine-tuned NMI values appear only in Figure 4. · 3.1 Protein Embeddings from the Pre-Trained…; 3.2 Protein Family Classification; 3.3.1 Conservative Scenario; 3.3.2 Aggressive Scenario

Methods, data and reproducibility

Swiss-Prot sequences are tokenized by byte pair encoding (10,000-token vocabulary) and used to pre-train a 5-layer Transformer encoder (d = 768, 12 attention heads, about 44M parameters) with masked language modeling in fairseq using the LAMB optimizer; a [CLS]-based output layer is then fine-tuned for family classification and for PPI prediction on concatenated sequence pairs.

  • UniProtKB/Swiss-Prot reviewed sequences for masked-language-model pre-training (mean tokenized length 129.6 tokens) · n = 450K unique sequences
  • Swiss-Prot proteins with curated protein family annotation, used for family classification fine-tuning · n = 313,214 unique protein sequences
  • Manually annotated human UniProt proteins, clustered and compared against annotated families · human · n = 9,151 proteins; 3,860 families
  • HIPPIE experimentally determined PPIs plus random pairs of human UniProt proteins as negatives (conservative scenario) · human · n = 600,563 interactions, including 411,428 negatives
  • HIPPIE PPIs plus unconnected HIPPIE protein pairs as negatives (aggressive scenario) · human · n = 536,545 interactions, including 275,402 negatives
  • Yeast interactions dataset used to probe small-data behaviour · yeast · n = 11,164 usable interactions
Methods & tools
fairseq, RoBERTa, LAMB optimizer, byte pair encoding (BPE), UniProtKB/Swiss-Prot, HIPPIE, PIPR, DeepFam, ProtVec, ProtDocVec, ProtFreqVec, k-means clustering
Assay conditions & compute
Pre-training took 18 hours on 4 NVIDIA V100 GPUs (compared with 128 V100 GPUs for 4 days reported for the Rives 2019 model); fine-tuning hardware and time are not stated.
Reproducibility gaps
The text links no code, trained weights or generated negative-example sets; no software versions are given (only the fairseq toolkit is named).
Limitations (6)
  • Negatives are sampled at random on the assumption that PPI networks are sparse; in the aggressive scenario some sampled negatives could be real interactions that were never recorded. · stated by the authors
  • On small datasets performance drops and the model overfits: on a yeast set with 11,164 usable interactions, train accuracy was 0.98 against test accuracy 0.77. · stated by the authors
  • The conservative-trained PPI model reaches only 0.59 accuracy on the aggressive set, which the authors take to mean it partly recognises proteins present in HIPPIE rather than interactions. · stated by the authors
  • Multi-class family classification assumes one family per protein; proteins belonging to more than one family were excluded from that classifier. · stated by the authors
  • Apart from a small yeast set mentioned only to show overfitting, the benchmarks are internal splits of Swiss-Prot and HIPPIE, with no independent external test set; the headline PPI comparisons train on only 20% of the data, and competitor scores are in tables and figures not given in the text. · noted in this summary
  • The 170-fold efficiency figure compares GPU time with a model trained on a different corpus for a different task (secondary structure), so it is not a matched accuracy comparison. · noted in this summary

Read this if you want a BERT-style protein language model pre-trained on 4 GPUs with BPE tokenization and fine-tuned for family classification and sequence-only PPI prediction, or need to see how negative sampling choices change PPI scores.

Citations
54

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISBhattachary '21Yang '24Wu '24Yang '22Szelogowski '23Brandes '22
71% semantic matchBioinformatics
Nicholas Bhattacharya, Neil Thomas, Roshan Rao+5 more·Pacific Symposium on Biocomputing (PSB)·Nov 28, 2021· 20 citations