Bilingual Language Model for Protein Sequence and Structure
Michael HeinzingeriD, Konstantin WeißenowiD, Joaquín Gómez SáncheziD, Adrian Henkel, Milot MirditaiD, Martin SteineggeriD, Burkhard RostiD
Abstract
Adapting large language models (LLMs) to protein sequences spawned the development of powerful protein language models (pLMs). Concurrently, AlphaFold2 broke through in protein structure prediction. Now we can systematically and comprehensively explore the dual nature of proteins that act and exist as three-dimensional (3D) machines and evolve as linear strings of one-dimensional (1D) sequences. Here, we leverage pLMs to simultaneously model both modalities by combining 1D sequences with 3D structure in a single model. We encode protein structures as token sequences using the 3Di-alphabet introduced by the 3D-alignment method Foldseek. This new foundation pLM extracts the features and patterns of the resulting "structure-sequence" representation. Toward this end, we built a non-redundant dataset from AlphaFoldDB and fine-tuned an existing pLM (ProtT5) to translate between 3Di and amino acid sequences. As a proof-of-concept for our novel approach, dubbed Protein structure-sequence T5 (ProstT5), we showed improved performance for subsequent prediction tasks, and for "inverse folding", namely the generation of novel protein sequences adopting a given structural scaffold ("fold"). Our work showcased the potential of pLMs to tap into the information-rich protein structure revolution fueled by AlphaFold2. ProstT5 paves the way to develop new tools integrating the vast resource of 3D predictions, and opens new research avenues in the post-AlphaFold2 era. Our model is freely available for all at https://github.com/mheinzinger/ProstT5.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 312
- FWCI
- 41.26
- Field percentile
- top 1%
- Assay
- Research Focus
- Approach
In-depth summary
Claude Sonnet 4.6 · xhigh effort · 7 Nov 2026 · from the full text of version 2 (24 Mar 2024) · CC BY 4.0
ProstT5, a ProtT5 fine-tuned to translate between amino acids and Foldseek 3Di tokens, predicts structure strings for remote homology search and performs inverse folding.
Structure predictions now exist for over 214 million UniProt sequences, but protein language models have mostly used sequence alone, and predicting structures for proteome-scale search is slow (48h on GPU for one proteome with ColabFold). This work trains one model to translate in both directions between amino acid sequences and 3Di structure tokens.
Key results
Shown3Di strings that ProstT5 predicts from sequence, used as Foldseek input, approach experimental-structure sensitivity for remote homology detection on SCOPe40 and far exceed sequence search.
Superfamily-level ROC-AUC: 0.45 for ProstT5 predicted 3Di, 0.49 for experimental 3Di, 0.06 for MMseqs2 sequence search. The two-layer CNN on the encoder reached 0.47. · 3600-fold faster structural remote homology…; Figure legends
ShownEncoder-plus-CNN prediction of 3Di tokens for a whole proteome takes seconds on GPU, versus hours of structure prediction. This is a speedup of over three orders of magnitude.
M. jannaschii proteome: 43.5s (±2s) on an Nvidia RTX A5000 GPU and 40min 6s (±4min 38s) on a MacBook M1 CPU, versus 48h for ColabFold structure prediction on GPU. · 3600-fold faster structural remote homology…; Proteome runtime benchmark
ShownReversing the translation (3Di to amino acids) generates diverse sequences that ESMFold predicts to fold like the native protein, though ProteinMPNN stays slightly ahead. A roundtrip 3Di check tracks sequence quality.
ProstT5 average lDDT 72 at as little as 21% PIDE to native, vs ProteinMPNN 77 and Native/ESMFold upper bound 78. Roundtrip accuracy vs lDDT: Spearman R 0.64. · Inverse Folding: creating new diverse protein…; Sampling from proteins’ Janus face
ShownProstT5 embeddings classify CATH levels better than sequence-only pLMs in unsupervised embedding-based annotation transfer. Amino acid and 3Di embeddings are complementary.
All ProstT5 input modes improved over ProtT5, ESM-1b and Ankh (Table 1). Concatenating amino acid and predicted-3Di embeddings beat either alone at all CATH levels. Numeric values are not in the text. · CATH classification of proteins
ShownStructure-sequence training helps structure-related tasks but slightly hurts function-related ones, so ProstT5 is not a general-purpose pLM. Concatenating ProtT5 and ProstT5(AA) embeddings partly compensates.
ProstT5(AA) beat ProtT5 on secondary structure Q3; Q3 approached 90% on NEW364 with experimental 3Di (circular). Predicted 3Di dropped below ProtT5. Location, binding and conservation declined. · Bilinguality improved structure encoding; ProstT5 is not a general-purpose pLM; Traditional embedding extraction
Methods, data and reproducibility
ProtT5 (3B parameters) was extended with 20 lower-case 3Di tokens and direction prefixes. It was trained with span denoising on both alphabets, then on bidirectional AA-3Di translation, and evaluated via embeddings, Foldseek search and ESMFold-based inverse folding.
- Training set from clustered AFDB (214M UniProt sequences, 52M then 18.8M clusters), filtered pLDDT>=70, length>=30, <=95% single 3Di token; both translation directions · n = 17M proteins (34M samples)
- Validation and test splits (cluster-disjoint representatives) used for sampling choice and inverse-folding evaluation · n = 474 proteins each
- SCOPe40 (SCOPe 2.01 at 40% identity) all-against-all remote homology benchmark
- Secondary structure: NetSurfP-2.0 training data; test sets CASP12, CASP14, NEW364
- Binding residues (DevSet1014, TestSet300), conservation (ConSurf10k), subcellular location (DeepLoc, setHARD)
- CATH v4.3 superfamily classification with a 20% non-redundant test set (EAT and contrastive learning)
- Methanocaldococcus jannaschii proteome (sequences shorter than 1000 AA) for runtime benchmark · Methanocaldococcus jannaschii · UP000000805
- Methods & tools
- ProtT5, ProstT5, Foldseek (3Di), MMseqs2, ESMFold, AlphaFold2 / AFDB, ProteinMPNN, ESM-2, Ankh, biotrainer, FLIP, DeepSpeed
- Code
- https://github.com/mheinzinger/ProstT5https://huggingface.co/Rostlab/ProstT5https://search.foldseek.com/
- Data & accessions
- https://huggingface.co/datasets/Rostlab/ProstT5Datasethttps://rostlab.org/~deepppi/prosst5_PDBs.tar
- Software & environment
- DeepSpeed stage-2, gradient accumulation, bf16 mixed precision, PyTorch2.0 torchInductor; Hugging Face generation configurations given for both translation directions.
- Assay conditions & compute
- Training on 8 Nvidia A100 (80GB) with 10 days for the 3Di-token stage and 6 days plus 20 days for translation. Human-proteome embeddings take around 35m on one RTX A6000 (48 GB).
- Reproducibility gaps
- Only example scripts are mentioned; full training code is not stated as released.
Limitations (6)
- Filtering AFDB for high pLDDT removes most disordered proteins and enriches short, well-structured, helical proteins, which may bias the model. · stated by the authors
- 3Di token classes are highly imbalanced (50% of residues in 2 of 20 tokens), which impaired training; all-helical proteins were often represented by a single token. · stated by the authors
- Pre-training on 3Di, which captures secondary structure, while evaluating on secondary structure and structural classification raises circularity and leakage concerns. CASP12 and NEW364 overlap AlphaFold2 training data. · stated by the authors
- Function-related tasks (subcellular location, binding, conservation) performed slightly worse than with ProtT5, so the model is not general-purpose. · stated by the authors
- Inverse folding is judged only by ESMFold predictions of generated sequences against AlphaFold2 structures; no experimental validation of the generated sequences is reported. · noted in this summary
- Remote homology evaluation is a single benchmark (SCOPe40), and the quantitative CATH results are in tables whose values are not in the text. · noted in this summary
Who should read:Read this if you want to predict 3Di structure tokens from sequence for fast remote homology search, or to try language-model inverse folding.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- AtlasFold: Protein structure prediction with metagenomic-scale language models
Seonghwan Seo, Hyeongwoo Kim, … Team KAIST·bioRxiv·Sep 7, 2026
- SPOT-1D-LM: Reaching Alignment-profile-based Accuracy in Predicting Protein Secondary and Tertiary Structural Properties without Alignment.
Jaspreet Singh, Kuldip K. Paliwal, … Yaoqi Zhou·bioRxiv·Oct 16, 2021
- SaProt: Protein Language Modeling with Structure-aware Vocabulary
Jin Su, Chenchen Han, … Fajie Yuan·bioRxiv·Oct 1, 2023
- Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Konstantin Weißenow, Michael Heinzinger, Burkhard Rost·StructurePeer-reviewed·May 23, 2022
- RemoteFoldSet: Benchmarking Structural Awareness of Protein Language Models
Zinnia Ma, Neville P. Bethel·bioRxiv·Sep 23, 2025
- ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention
Mingchen Li, Yang Tan, … Liang Hong·neural information processing systemsPeer-reviewed·Jan 1, 2024