henhaobio
Sign in

BioinformaticsBioinformaticsPeer-reviewed journal article

ProteinBERT: A universal deep-learning model of protein sequence and function

Nadav BrandesiD, Dan OferiD, Yam Peleg, Nadav RappoportiD, Michal Linial BiD

PDFDOIPreprintCode
In-depth full-text digest available · Jump to digest ↓

Abstract

Self-supervised deep language modeling has shown unprecedented success across natural language tasks, and has recently been repurposed to biological sequences. However, existing models and pretraining methods are designed and optimized for text analysis. We introduce ProteinBERT, a deep language model specifically designed for proteins. Our pretraining scheme consists of masked language modeling combined with a novel task of Gene Ontology (GO) annotation prediction. We introduce novel architectural elements that make the model highly efficient and flexible to very large sequence lengths. The architecture of ProteinBERT consists of both local and global representations, allowing end-to-end processing of these types of inputs and outputs. ProteinBERT obtains state-of-the-art performance on multiple benchmarks covering diverse protein properties (including protein structure, post translational modifications and biophysical attributes), despite using a far smaller model than competing deep-learning methods. Overall, ProteinBERT provides an efficient framework for rapidly training protein predictors, even with limited labeled data. Code and pretrained model weights are available at https://github.com/nadavbra/protein_bert.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (25 May 2021) · CC BY 4.0

A ~16M-parameter protein language model, pretrained on ~106M UniRef90 proteins by denoising sequences and Gene Ontology annotations, rivals larger TAPE models using linear-cost global attention.

Prior protein language models (e.g. ESM, TAPE-Transformer, UniRep) largely reuse natural-language architectures and pretraining tasks, although proteins lack word-like units, vary greatly in length and couple distant residues via 3D structure. ProteinBERT adds a protein-specific pretraining task (Gene Ontology annotation recovery) and a local/global architecture whose compute grows linearly with length, so a far smaller model handles very long proteins intact.

Key results

  1. ShownProteinBERT is much smaller than the TAPE-evaluated Transformer and LSTM models, yet it matches or beats them on the four TAPE benchmarks.

    Tasks: secondary structure, remote homology, fluorescence, stability. Baselines (TAPE BERT Transformer, LSTMs) have ~38M parameters vs ~16M for ProteinBERT. Per-task metrics are in Table 2 and Supplementary Table S1, which are not reproduced in the extracted text. · ProteinBERT achieves state-of-the-art results on…; Deep-learning architecture; Tables

  2. ShownPretraining strongly improves downstream accuracy. Compared with training from scratch, it helped every TAPE task. The harder tasks (secondary structure, remote homology) kept improving with more pretraining without plateauing, while some tasks gained nothing.

    With vs without pretraining comparison on the TAPE tasks (Table 2); models fine-tuned from 371 pretraining snapshots showed unsaturated gains for secondary structure and remote homology. Over ~670M records (~6.4 epochs), language-modeling training loss kept decreasing while GO-annotation loss saturated. · Self-supervised pretraining on protein sequences…; Pretraining improves protein modeling; ProteinBERT achieves state-of-the-art results on…; Figure legends

  3. SuggestedThe same weights work across sequence lengths. Accuracy on proteins encoded beyond 512 tokens is usually only modestly lower, and some longer bins score higher, which the authors take as a hint that such shifts may stem from other factors like task difficulty.

    Evaluated on 4 of 9 benchmarks having at least 25 test records in proteins longer than 512 tokens; higher scores seen for 16,384-token sequences in Major PTMs and 1,024-token sequences in Neuropeptide cleavage. No numeric differences are given in the text. · ProteinBERT generalizes across protein lengths; Figure legends

  4. ShownProteinBERT is compact and fast: a single GPU sufficed for pretraining, and adapting it to a new task takes minutes.

    ~16M parameters vs ~38M (TAPE Transformer), ~110M (BERT-base) and ~650M (ESM-1b). Pretraining: 280 records/s on one Nvidia Quadro RTX 5000 for 28 days (~670M records). Fine-tuning: ~14 minutes on one GPU, averaged over the nine benchmarks. Global attention makes compute and memory scale linearly with length. · Self-supervised pretraining on protein sequences…; Supervised fine-tuning on protein benchmarks; Deep-learning architecture

  5. SuggestedGlobal attention produces compact 2D attention maps that can be inspected directly. Fine-tuning for signal peptide prediction mostly reshaped the final (6th) global attention layer.

    All 24 heads examined for two signal-peptide test proteins (mouse Hebp1, Arabidopsis GAMMACAL2) before and after fine-tuning: block 3 head #3 favored sequence starts and head #2 the middle/end; block 6 head #1 shifted toward sequence starts. Authors note values can vary between training runs. · Understanding global attention; Discussion; Figure legends

Methods, data and reproducibility

Denoising-autoencoder transformer with parallel local (per-residue, 128 channels; narrow and dilated 1D convolutions) and global (protein-level, 512-dim) tracks over 6 blocks, coupled by broadcast layers and a linear-cost global attention; pretrained to recover corrupted sequences and Gene Ontology annotations, then fine-tuned with an added output layer.

  • UniRef90 representative proteins with UniProtKB sequences and GO annotations (8,943 terms seen at least 100 times), used for pretraining · multiple species (spanning the tree of life) · n = ~106M proteins; ~46M carry at least one of the GO terms
  • TAPE benchmarks: secondary structure, remote homology, fluorescence and stability prediction · n = four benchmarks
  • Benchmarks newly introduced by the authors (e.g. signal peptide, Major PTMs, Neuropeptide cleavage), defined in Supplementary Methods · n = five benchmarks
  • Pretraining proteins with at least 40% similarity to any benchmark test record (found by BLASTP); their GO labels were withheld · multiple species · n = ~600K sequences
  • Signal peptide test-set proteins used to visualize global attention: Hebp1 and GAMMACAL2 · mouse; Arabidopsis · n = two proteins
Methods & tools
ProteinBERT (Python, TensorFlow Keras), UniProtKB / UniRef90, Gene Ontology (GO), BLASTP (default settings), TAPE benchmark suite, TAPE BERT Transformer (baseline), TAPE LSTM models (baseline), BERT-base and ESM-1b (parameter-count comparison only)
Software & environment
Python package implemented in TensorFlow's Keras; downloads pretrained weights automatically and supports fine-tuning and evaluation. No library or framework versions are stated.
Assay conditions & compute
Single Nvidia Quadro RTX 5000 GPU: pretraining at 280 protein records/s for 28 days over ~670M records; fine-tuning ~14 minutes per benchmark on one GPU (average over nine).
Reproducibility gaps
Benchmark definitions, the five new benchmarks and full per-task metrics are in Supplementary Methods/Table S1, absent from this text; datasets are regenerated via repository scripts, with no archival accession or pinned versions.
Limitations (5)
  • Attention maps depend on the trained weights, which differ between runs. The authors report that fine-tuning is mostly consistent but small run-to-run differences occur, so head-level interpretations are not fixed. · stated by the authors
  • The authors caution that performance shifts across sequence lengths may stem from other factors, such as longer proteins being inherently harder for tasks like secondary structure, so per-length differences should be read cautiously. · stated by the authors
  • Compute and model size were modest (one GPU, ~16M parameters), and language-modeling loss had not saturated after ~6.4 epochs. The authors expect larger, longer-trained versions to do better, so the reported results are likely not the model's ceiling. · stated by the authors
  • Test-set leakage is only partly controlled: the ~600K pretraining proteins with at least 40% similarity to test records lost their GO labels but apparently stayed in the language-modeling corpus. · noted in this summary
  • In the text, accuracy comparisons are only against TAPE's Transformer and LSTMs. ESM-1b is compared by parameter count, not accuracy, and no external baselines are described for the five new benchmarks. · noted in this summary

Read this if you want a small protein language model you can fine-tune in minutes on structure, post-translational modification or biophysical tasks, or are designing linear-cost attention for very long sequences.

Citations
1,009
FWCI
62.07
Field percentile
top 1%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISDuan '25Capel '22Elnaggar '21Heinzinger '19Appel '26Chen '23
74% semantic matchBiochemistry
Haonan Duan, Marta Skreta, Leonardo Cotta+4 more·Nature Machine Intelligence·Aug 18, 2025· 11 citations