henhaobio
Sign in

bioRxivSystems BiologyPreprint

Learning inverse folding from millions of predicted structures

Chloe HsuiD, Robert VerkuiliD, Jason J. LiuiD, Zeming LiniD, Brian HieiD, Tom SercuiD, Adam K. LereriD, Alexander W. RivesiD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

We consider the problem of predicting a protein sequence from its backbone atom coordinates. Machine learning approaches to this problem to date have been limited by the number of available experimentally determined protein structures. We augment training data by nearly three orders of magnitude by predicting structures for 12M protein sequences using AlphaFold2. Trained with this additional data, a sequence-to-sequence transformer with invariant geometric input processing layers achieves 51% native sequence recovery on structurally held-out backbones with 72% recovery for buried residues, an overall improvement of almost 10 percentage points over existing methods. The model generalizes to a variety of more complex tasks including design of protein complexes, partially masked structures, binding interfaces, and multiple states.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (6 Sept 2022) · CC BY-NC-ND 4.0

Adding 12 million AlphaFold2-predicted structures to training lifts native sequence recovery on held-out backbones to 51.6%, versus 42.2% for the best experimental-only model; likelihoods score mutation effects zero-shot.

Deep inverse-folding models learn from experimental structures, which cover under 0.1% of known protein sequence space, so they are data-limited. This work treats AlphaFold2 predictions for 12 million UniRef50 sequences as extra training data, in the spirit of back-translation in machine translation, and shows that larger models, unlike the previous best GVP-GNN, can exploit it.

Key results

  1. ShownAdding AlphaFold2-predicted structures to training lifted native sequence recovery on topology-held-out CATH backbones, with the GVP-Transformer reaching 51.6%, 9.4 points above the best model trained on CATH alone.

    Recovery with vs without predicted structures: GVP-Transformer 51.6% vs 38.3%, GVP-GNN-large 50.8% vs 39.2%; best CATH-only model GVP-GNN 42.2%; 1797 test chains. Recovery is 72% for core and 39% for surface residues, and conclusions hold on a stricter 223-protein TM-score-filtered test set. · 3.1. Fixed backbone protein design; B. TM-score-based test set; Training and evaluation data

  2. ShownExploiting predicted structures requires larger models: the small GVP-GNN gets worse with them, and training on predicted structures alone is substantially worse than training on CATH alone.

    GVP-GNN (1M parameters) fell from 42.2% to 38.6% recovery with predicted data, and scaling it to 21M parameters on CATH alone overfit (42.2% to 39.2%). Gains grew substantially up to 1 million predicted structures (75 times CATH), with diminishing returns beyond; 0.1 Å coordinate noise slightly helped the GVP-Transformer. · 3.1. Fixed backbone protein design; Predicted structures; Model performance when trained only on predicted…; Noise on AlphaFold2-predicted backbone…

  3. ShownAlthough trained only on single chains, the models carry over to harder design settings: protein complexes, partially masked backbones, two-state design and a structurally remote target.

    Full complex coordinates (CATH 4.3 test complexes up to 1000 residues) substantially improved perplexity over single-chain input; the GVP-Transformer stayed moderate on masked spans where GVP-GNN-large fell to background beyond 5 residues; two-state design lowered perplexity at flexible residues in 87 PDBFlex proteins; SARS-CoV-2 RBD recovery reached 53.6%. · Protein complexes; Partially-masked backbones; Multiple conformation; 3.1. Fixed backbone protein design

  4. ShownModel likelihoods ranked SARS-CoV-2 RBD mutations by ACE2-binding affinity zero-shot, better than sequence-only language models, and this depended on structural information at the interface.

    Spearman 0.69 with measured binding for receptor-binding-motif mutations given RBD plus ACE2 coordinates (PDB 6M0J); weaker without ACE2 and no correlation once the motif (69 of 195 residues) was masked; outperformed ESM-1b, ESM-1v and ESM-MSA-1b masked-marginal scores. · Binding affinity; Predicting RBD-ACE2 binding affinity

  5. ShownZero-shot likelihood scores also tracked other fitness readouts (AAV capsid insertions, mini-protein stability, complex stability), though predicted-structure training did not help complex stability.

    AAV packaging (FLIP, 7 subsets): GVP-Transformer beat ESM-1v on 6 of 7, and predicted-structure models gained more over GVP-GNN as variants moved further from wild type. Mini-proteins: predicted-structure training improved stability correlation on 8 of 10 folds. SKEMPI (Atom3D): comparable to the best supervised transfer-learning method, with no gain from predicted structures. · Sequence insertions; Predicting insertion effects on AAV; De novo mini-proteins; Complex stability

Methods, data and reproducibility

Backbone N, Cα and C coordinates are encoded with rotation-invariant geometric vector perceptron (GVP) layers and decoded autoregressively to sequence, either by a GVP-GNN or by GVP layers feeding a 142M-parameter encoder–decoder transformer (GVP-Transformer). Each epoch mixes ~16K CATH chains with a 10% sample of AlphaFold2 predictions for 12 million UniRef50 sequences (1:80), with span masking and coordinate noise.

  • CATH v4.3 40% non-redundant chains of up to 500 residues, split 80/10/10 by topology · n = 16,153 train, 1457 validation, 1797 test chains
  • AlphaFold2 (Model 1) structures predicted for UniRef50 release 202001 sequences under 500 residues, with validation/test topologies removed via Gene3D HMMs · n = 12 million
  • Stricter test set with no TM-score ≥ 0.5 match to training (foldseek TMalign) · n = 223 proteins
  • PDBFlex proteins with two distinct conformations, from the CATH test split · n = 87 proteins
  • SARS-CoV-2 RBD deep mutational scan of ACE2-binding affinity, plus RBD open and closed states · SARS-CoV-2 (with human ACE2) · PDB 6M0J; 6XRA; 6VXX
  • AAV capsid insertion and substitution variants (FLIP, 7 subsets) scored on the wild-type backbone · adeno-associated virus · n = more than 200,000 variants · PDB 1LP3
  • Deep mutational scans of de novo designed mini-proteins measuring stability · n = 10 folds
  • SKEMPI binding free-energy changes, as the Atom3D classification task
Methods & tools
AlphaFold2, MSA Transformer, GVP-GNN, GVP-GNN-large, GVP-Transformer, Gene3D / hmmsearch, hhblits, Foldseek (TMalign), ESM-1b, ESM-1v, ESM-MSA-1b, fairseq
Reagents & environment
Named components: fairseq incremental decoding for the transformer decoder, gvp-pytorch for GVP-GNN, PyTorch Profiler; AlphaFold2 Model 1 (CASP14 weights) with 3 recycles and Amber relaxation. No software versions or containers are given.
Assay conditions & compute
Sampling speed was profiled on a Quadro RTX 8000 GPU (48GB); AlphaFold2 inputs were reduced (single model, no metagenomic MSAs or templates) to fit the compute budget. Training hardware and time are not stated.
Reproducibility gaps
This text has no code or data availability statement, so release of the models or the 12 million predicted structures cannot be confirmed from it; detailed hyperparameters are referenced but not included here.
Limitations (6)
  • Low perplexity and high recovery do not guarantee structural similarity, and sampled sequences can collapse into runs of a single amino acid. · stated by the authors
  • Training on predicted structures alone is substantially worse than on experimental structures alone, and AlphaFold2 coordinates carry sub-ångström artifacts that may reveal residue identity, which is why noise is added during training. · stated by the authors
  • Predicted structures improved perplexity but not single-point complex-stability prediction on SKEMPI, which the authors read as a limit of single-mutation benchmarks. · stated by the authors
  • The standard CATH topology split still allows some high TM-score train–test matches; results on the stricter TM-score-filtered set appear only in a supplement. · stated by the authors
  • No designed sequence was folded in silico or tested experimentally here, so whether designs adopt the target backbone is not established. · noted in this summary
  • Zero-shot affinity ranking rests on one deep mutational scan (SARS-CoV-2 RBD–ACE2; SKEMPI enters only as a binary stability-change task), and complex and PDBFlex two-state gains are reported only as perplexity changes. · noted in this summary

Read this if you design sequences for fixed backbones or score mutations with structure-conditioned likelihoods and want evidence on training with predicted structures.

Citations
438

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISYang '22Chowdhury '22Ahdritz '24McPartlon '22Hong '26Moffat '21
79% semantic matchBioengineering
K. K. Yang, H. Yeh, N. Zanichelli·bioRxiv·May 25, 2022