henhaobio
Sign in

bioRxivSynthetic BiologyPreprint

Evaluating Protein Transfer Learning with TAPE

Roshan RaoiD, Nicholas BhattacharyaiD, Neil ThomasiD, Yan DuaniD, Xi CheniD, John CannyiD, Pieter AbbeeliD, Yun S. SongiD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to real-life scenarios. We bench-mark a range of approaches to semi-supervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that self-supervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by self-supervised pretraining still lag behind features extracted by state-of-the-art non-neural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https://github.com/songlab-cal/tape.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 1 (20 Jun 2019)

Across five protein tasks with evolution-aware splits, self-supervised pretraining helped nearly every model on every task, yet alignment-based features still beat learned embeddings on secondary structure and contact prediction.

Self-supervised protein representation methods had been evaluated on different tasks, test sets and splitting schemes, so their relative merits could not be compared. TAPE fixes five supervised tasks (secondary structure, contacts, remote homology, fluorescence, stability) with splits that hold out distant sequences or probe mutational neighbourhoods, and compares recurrent, convolutional and attention models against one-hot and alignment features under identical fine-tuning.

Key results

  1. ShownSelf-supervised pretraining improved downstream performance for almost all models on all five tasks, in some cases more than doubling it.

    Pretrained vs non-pretrained LSTM, Transformer, ResNet and the Bepler et al. and Alley et al. models; per-task scores are in a results table not reproduced in this text. On stability, the best pretrained model called mutations stabilising vs destabilising (parent as decision boundary) with 70% accuracy vs 68% for the best non-pretrained model. · Abstract; 6 Results

  2. ShownLearned embeddings still trail alignment-based inputs and published alignment-using methods on structure-related tasks.

    Best TAPE model vs prior methods: CB513 secondary structure 75% vs 85% accuracy (NetSurfP-2.0); CASP12 contact precision at L/5 0.49 vs 0.69 (RaptorX); remote homology 26% vs 41% accuracy (DeepSF). Within TAPE's controlled setup, alignment features beat learned ones on secondary structure and contacts, whereas learned features did better on remote homology. · Comparison to state of the art; 1 Introduction; A.6.1 Secondary Structure Results

  3. ShownNo architecture is best across tasks; rankings flip between structure and engineering tasks, so a single-task benchmark would mislead.

    The Transformer was the weakest of the three new architectures on secondary structure but the strongest on fluorescence and stability; the ResNet tied the LSTM on secondary structure yet reached Spearman ρ 0.21 vs the LSTM's 0.67 on fluorescence. · Need for multiple benchmark tasks; 8 Future Work

  4. ShownLanguage-model perplexity is a poor guide to downstream usefulness, and language-modelling accuracy falls on Pfam families withheld from pretraining.

    Models with lower perplexity on the random-split and held-out sets did not necessarily do better on downstream tasks, replicating Rives et al.; held-out-family accuracy was consistently below random-split accuracy (values in a table not reproduced here). · 6 Results

  5. ShownPretraining helps the best models tell bright from dark GFP variants, though many are still misclassified, and low Spearman ρ within the dark mode suggests they cannot rank variants there.

    Fluorescence labels are bimodal; pretraining reduced mean-squared error in both bright and dark modes, but Spearman ρ within the more diverse dark mode stayed low and many proteins were still misclassified. · 6 Results; A.6.4 Fluorescence Results

Methods, data and reproducibility

LSTM, Transformer and dilated ResNet models of roughly matched size (38M parameters for the Transformer), plus the Bepler et al. and Alley et al. models, were pretrained on Pfam by next-token or masked-token prediction, then fine-tuned with one fixed downstream architecture per task on five tasks (8 thousand to 50 thousand training examples) and compared with one-hot and alignment-based features.

  • Pfam protein domains for pretraining, with a test set of fully held-out families and clans (about 1% of the data) and a random 95/5% split of the rest · n = thirty-one million protein domains
  • Secondary structure: Klausen et al. training/validation data filtered at 25% sequence identity; test sets CB513, CASP12 and TS115
  • Contact prediction: ProteinNet splits filtered at 30% sequence identity, evaluated on the CASP12 test set
  • Remote homology: Hou et al. data derived from SCOP 1.75 with whole superfamilies held out; 1195 fold classes
  • Fluorescence: deep mutational scanning of green fluorescent protein (Sarkisyan et al.); train within Hamming distance 3 of the parent, test at distance 4-15
  • Stability: Rocklin et al. designed proteins from four design rounds; test set of seventeen Hamming-distance-1 neighbourhoods around top candidates
Methods & tools
TAPE, Pfam, ProteinNet, SCOP 1.75, Transformer, LSTM, dilated ResNet, Bepler et al. model, Alley et al. mLSTM, NetSurfP-2.0, RaptorX-Contact, DeepSF
Assay conditions & compute
Self-supervised models: four NVIDIA V100 GPUs on AWS for one week each. Supervised models: two NVIDIA Titan Xp GPUs (two Titan RTX for contact prediction) until 10 epochs passed without validation improvement.
Reproducibility gaps
The paper says all code, data, splits, features and per-model batch sizes are in a public repository, but the link is not in this text; no software versions or environment files are given.
Limitations (5)
  • Supervised heads were not hyperparameter-tuned, and baselines drop parts of published pipelines: NetSurfP-2.0 without multitask training (worth one to two percent), and PSSMs instead of RaptorX's CCMpred-based features for contacts. · stated by the authors
  • Sequence-identity filtering shrank the Bepler et al. supervised pretraining set by 75%, which likely weakened that procedure. · stated by the authors
  • Alignment features are not used for the fluorescence and stability tasks because all variants would get the same alignment profile, and these splits are new, so no prior state of the art exists for them. · stated by the authors
  • Contact precision decays from short- to long-range contacts for all models, unlike many recent CASP methods, and stability prediction success varies strongly by protein topology. · stated by the authors
  • The three new architectures were parameter-matched around 38M and all models were pretrained for one week on Pfam with two self-supervised objectives only, so the rankings may not carry over to larger models or other corpora. · noted in this summary

Read this if you build or evaluate protein language models and want standard tasks with evolution-aware splits, or need to know where pretrained embeddings still lose to alignment features.

Citations
719

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISShen '21Capel '22Ma '23Zhou '23Shen '26Luo '20
70% semantic matchBioinformatics
Hongyu Shen, Layne C. Price, Mohammad Taha Bahadori+1 more·bioRxiv·Feb 18, 2021· 15 citations