henhaobio
Sign in

bioRxivBioinformaticsPreprint

SaProt: Protein Language Modeling with Structure-aware Vocabulary

Jin SuiD, Chenchen HaniD, Yuyang ZhouiD, Junjie Shan, Xibin ZhouiD, Fajie YuaniD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

AO_SCPLOWBSTRACTC_SCPLOWLarge-scale protein language models (PLMs), such as the ESM family, have achieved remarkable performance in various downstream tasks related to protein structure and function by undergoing unsupervised training on residue sequences. They have become essential tools for researchers and practitioners in biology. However, a limitation of vanilla PLMs is their lack of explicit consideration for protein structure information, which suggests the potential for further improvement. Motivated by this, we introduce the concept of a "structure-aware vocabulary" that integrates residue tokens with structure tokens. The structure tokens are derived by encoding the 3D structure of proteins using Foldseek. We then propose SaProt, a large-scale general-purpose PLM trained on an extensive dataset comprising approximately 40 million protein sequences and structures. Through extensive evaluation, our SaProt model surpasses well-established and renowned baselines across 10 significant downstream tasks, demonstrating its exceptional capacity and broad applicability. We have made the code1, pre-trained model, and all relevant materials available at https://github.com/westlake-repl/SaProt.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 5 (19 Apr 2024)

Pairing each residue with its Foldseek 3D-structure token and pre-training an ESM-2-sized model on about 40 million AlphaFold2 structures beat ESM-2 and structure-based baselines on variant-effect and function tasks.

Sequence-only protein language models such as ESM ignore explicit structure, while earlier structure-based models used few structures (GearNet: around 800 thousand AlphaFold2 predictions) or served single tasks such as inverse folding. Encoding structure as per-residue tokens lets a standard Transformer train on about 40 million predicted structures, avoiding the failure the authors saw when graph or attention-bias models were trained on predicted structures.

Key results

  1. ShownIn zero-shot mutational effect prediction, SaProt outperformed every residue sequence-based and structure-based baseline on ProteinGym and ClinVar, including ESM-2 with the same architecture and training examples, and stayed best among language models when multiple sequence alignment (MSA) information was added.

    Baselines: ESM-1b, ESM-1v, ESM-2 650M, Tranception L, MIF-ST and ESM-IF, with EVE and MSA Transformer also reported. Metrics: Spearman's rank correlation on ProteinGym, AUC on ClinVar. SaProt also beat ESM-2 15B and ProstT5 on the zero-shot tasks. The scores are in result tables not included in this text. · 4.1.3 Results; 4.1.2 Baselines & Evaluation; D.2.3 Additional Comparison; A Comparison With ProstT5

  2. ShownOn supervised protein-level tasks SaProt beat ESM-2 on every task, with the clearest gains on Thermostability, HumanPPI, Metal Ion Binding and DeepLoc, and beat the structure models GearNet and MIF-ST by a substantial margin.

    Tasks: FLIP Thermostability, Metal Ion Binding (re-split at 30% sequence identity), DeepLoc, EC and GO annotation, and HumanPPI, with all models fine-tuned under one shared hyper-parameter set. Swapping SaProt into ESM-GearNet (SaProt-GearNet) beat ESM-GearNet, though ensembles did not always beat their single models. Values are not in this text. · 4.2.3 Results; 4.2.1 Datasets; D.3.2 Dataset Split; D.3.3 Training Details

  3. ShownSaProt holds more explicit structural information than ESM-2: with frozen backbones it predicts residue contacts markedly better, and its embeddings separate alpha from beta proteins where ESM-2's are intertwined.

    TAPE contact prediction with only the contact head fine-tuned, evaluated on PDB structures. With all structure tokens masked, SaProt (Residue-only) falls below ESM-2 but keeps some contact ability; randomly replacing structure or residue tokens lowers zero-shot accuracy. t-SNE on SCOPe filtered to below 40% identity. Values are not in this text. · 5.1 Awareness Of Protein Structure; 5.3 Visualization

  4. ShownResults depend on the source of the structure tokens: SaProt trained on AlphaFold2 structures works about as well on PDB inputs, a PDB-tuned variant prefers PDB inputs, and tokens from ESMFold fall short of AlphaFold2 tokens.

    SaProt-PDB (further pre-trained on 60,000 PDB structures): the authors cite a zero-shot 0.454 vs 0.423 to show the cost of strongly mismatched training and test structures (task and metric not stated here). With ESMFold tokens SaProt is better than or comparable to ESM-2. Fine-tuned with all structure tokens masked, it is on par with ESM-1b and ESM-2 on Fluorescence, Stability, beta-lactamase and AAV. · 5.2 PDB versus AlphaFoldDB; E.4 Evaluation With ESMFold; E.3 Residue Sequence-only Fine-tuning

  5. SuggestedPre-training directly on predicted structures with a graph network or with structure biases in attention failed: loss fell on AlphaFold2 structures but not on real PDB structures, and downstream results were poor.

    Two exploratory pre-training runs (a graph neural network, and an Evoformer-inspired model on an ESM-2 35M base); the authors suggest the models picked up traces of AlphaFold2 predictions. Predicting only residues, with structure tokens kept as context, beat predicting whole structure-aware tokens on ClinVar, ProteinGym and DeepLoc. Values are not in this text. · 3.1 Preliminary Analysis; E.1.2 Experimental Results For Three Models; Masking Strategy 2

Methods, data and reproducibility

Each residue becomes one token pairing its amino acid with the Foldseek 3Di state at that position (441-token vocabulary, low-confidence regions with structure masked), and an ESM-2 650M architecture is pre-trained by masked language modeling on about 40 million AlphaFold2 structures to predict residues. It is tested zero-shot and by fine-tuning on 10 tasks.

  • AlphaFoldDB predicted structures for ESM-2-filtered UniProt sequences, encoded as Foldseek 3Di tokens (pre-training) · n = about 40 million structures
  • PDB structures for continued pre-training of SaProt-PDB · n = 60,000 structures
  • ProteinGym deep mutational scanning substitution assays (zero-shot)
  • ClinVar variants as curated in EVE, 1 Gold Star or higher (zero-shot) · human
  • Supervised benchmarks: FLIP Thermostability and AAV, Metal Ion Binding, DeepLoc, DeepFRI EC and GO, PEER HumanPPI and beta-lactamase, TAPE Fluorescence, Stability and contact prediction
  • SCOPe non-redundant set for embedding visualization
Methods & tools
SaProt, Foldseek (3Di structure tokens), AlphaFold2 / AlphaFoldDB, ESMFold, ESM-2 (650M, 15B), ESM-1b, ESM-1v, Tranception L, MIF-ST, ESM-IF, EVE, MSA Transformer, GearNet / ESM-GearNet, ProstT5
Assay conditions & compute
Pre-training ran 3 months on 64 NVIDIA 80G A100 GPUs, a cost similar to ESM-1b, with mixed precision, about 3M steps, batch size 512 and sequences truncated to 1024 tokens.
Reproducibility gaps
Code, pre-trained weights (including SaProt-PDB) and datasets are said to be openly available, but the link was lost in extraction. Training hyper-parameters are listed, software versions are not, and every result table is missing from this text.
Limitations (6)
  • The vocabulary relies heavily on Foldseek, which balances search speed against encoding accuracy, so the representation could still be improved. · stated by the authors
  • Compute limits kept the model at 650M parameters, which may be below its best capacity. · stated by the authors
  • Foldseek tokens can be inaccurate, and because AlphaFold2 cannot resolve single-mutation structural changes, all variants of a protein are given the wild-type structure. · stated by the authors
  • When structure tokens are absent at fine-tuning time, SaProt shows no clear advantage over ESM models on mutational fitness datasets. · stated by the authors
  • Proteins without AlphaFoldDB structures, and ProteinGym or ClinVar proteins longer than 1024 residues, were dropped from evaluation, so results cover only proteins with predicted structures. · noted in this summary
  • Baselines were fine-tuned with one shared hyper-parameter set rather than tuned individually, which may understate some of them. · noted in this summary

Read this if you use protein language model embeddings for variant-effect or function prediction and want to know whether adding Foldseek structure tokens from predicted structures helps.

Citations
244

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISPourmirzaei '24Li '24Frolova '24Heinzinger '24Zhang '25Wang '24
83% semantic matchBioinformatics
Mahdi Pourmirzaei, Farzaneh Esmaili, Mohammadreza Pourmirzaei+2 more·bioRxiv·May 31, 2024· 9 citations