henhaobio
Sign in

bioRxivSynthetic BiologyPreprint

Language models generalize beyond natural proteins

Robert VerkuiliD, Ori KabeliiD, Yilun DuiD, Basile I. M. WickyiD, Lukas F. MillesiD, Justas DauparasiD, David A. BakeriD, Sergey OvchinnikoviD, Tom SercuiD, Alexander W. RivesiD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

Learning the design patterns of proteins from sequences across evolution may have promise toward generative protein design. However it is unknown whether language models, trained on sequences of natural proteins, will be capable of more than memorization of existing protein families. Here we show that language models generalize beyond natural proteins to generate de novo proteins. We focus on two protein design tasks: fixed backbone design where the structure is specified, and unconstrained generation where the structure is sampled from the model. Remarkably although the models are trained only on sequences, we find that they are capable of designing structure. A total of 228 generated proteins are evaluated experimentally with high overall success rates (152/228 or 67%) in producing a soluble and monomeric species by size exclusion chromatography. Out of 152 experimentally successful designs, 35 have no significant sequence match to known natural proteins. Of the remaining 117, sequence identity to the nearest sequence match is at median 27%, below 20% for 6 designs, and as low as 18% for 3 designs. For fixed backbone design, the language model generates successful designs for each of eight experimentally evaluated artificially created fixed backbone targets. For unconstrained generation, sampled proteins cover diverse topologies and secondary structure compositions, and have high experimental success rate (71/129 or 55%). The designs reflect deep patterns linking sequence and structure, including motifs that occur in related natural structures, and motifs that are not observed in similar structural contexts in known protein families. The results show that language models, though only trained on sequences, learn a deep grammar that enables the design of protein structure, extending beyond natural proteins.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 1 (22 Dec 2022) · CC BY-NC-ND 4.0

A sequence-only protein language model, sampled under a structure-compatibility energy, produced designs of which 152 of 228 tested behaved as soluble monomers, some with no detectable UniRef90 match.

Whether a model trained only on evolutionary sequence data can design proteins outside the families it memorised was open, and earlier experimentally validated sequence-based design stayed within natural families, above 30% identity to natural proteins. Here the same language model used for structure representation is turned into a generator for artificially designed backbones and for unconstrained sampling, with wet-lab testing of hundreds of designs rather than in silico scores alone.

Key results

  1. ShownIn a matched ablation, removing the language model collapses experimental success: with AlphaFold's distogram supplying the structure term and no sequence model, almost all designs were insoluble.

    Four de novo targets, 200 designs per method, top 5 per target tested with no further filtering: 95% (19/20) of language-model designs succeeded by size exclusion chromatography against 5% (1/20) without it, and 0/20 when an n-gram prior was added. Language-model perplexity separated success from failure; AlphaFold designs scored 10.6 to 13.1 against 6.7 for de novo target sequences. · Language models design sequences that fold to de…; Round 2 (LM vs. no-LM) (64x)

  2. ShownGiven artificially designed backbones the model writes sequences that express and behave as folded monomers, succeeding on every backbone tested, including an idealised TIM-barrel.

    Of 79 fixed backbone designs across 6 de novo targets, 97% (77/79) were soluble, 78% (62/79) passed size exclusion chromatography and 39% (31/79) were monodisperse; successes covered all 8 backbones tested overall, including length 182 TIM-barrel 6WVS. In silico across 39 targets, the best 10 of 200 designs had median RMSD < 2.5A for 84% (33/39) of targets. · Language models design sequences that fold to de…

  3. ShownSampling sequence and structure jointly, without a target backbone, produces varied folds that also survive the bench at a rate above half.

    25,000 samples at length 100 fell into 7,663 clusters at a TM-score threshold of 0.75, with 52% mostly alpha-helical, 22% mostly beta and 28% mixed, and median pLDDT 84.49. Of these, 20% passed in silico filters, 129 were expressed and 55% (71/129) succeeded; 96% were soluble and 30% monodisperse. · Language models generate novel structures and…; AlphaFold and Rosetta Quality Filters

  4. ShownSuccessful designs are not retrieved training sequences: a sizeable share have no significant hit in the corpus the model was trained on, and matched hits are often a different predicted fold.

    Of 152 experimental successes, 35 had no significant sequence match; the other 117 reached median 27% identity to their nearest match, with 6 designs under 20% and a floor of 18% in 3 cases. Among free generations, 31 of 49 designs distant in both sequence and predicted structure succeeded (given as 63% in the main text and 67% in the appendix), and 16 of those 31 had no significant match. · Abstract; Language models design sequences that fold to de…; Language models generate novel structures and…

  5. SuggestedDesigns carry recognisable structural grammar, including buried hydrogen-bond networks placed in sequence contexts where no similar natural sequence or structure shows that motif.

    Motifs include proline-induced helix bending, periodic glycines in beta-strands, helix capping, and networks of four or more buried polar or charged residues (a salt bridge in F129, a pi-cation bond in F025, a T-shaped pi-pi interaction in F030). For F129 and F092 the motif was absent from aligned positions of all jackhmmer and Foldseek hits; F030's closest natural match shares 26% identity. · Language models materialize deep patterns of…; A.5.4. Motif ANALYSIS

Methods, data and reproducibility

ESM2_650M scores sequences by pseudo-likelihood, while an affine projection from its attention maps to binned inter-residue distances scores structure given sequence. Fixed backbone design samples by Markov chain Monte Carlo with annealing under an energy joining both terms and an n-gram prior; free generation alternates structure and sequence sampling by blocked Gibbs. AlphaFold is the oracle.

  • De novo target set: structurally validated artificial protein structures from the PDB spanning alpha-bundle, beta-barrel, NTF2 and Rossman folds, lengths 67 to 184 · none (human-designed proteins) · n = 39 structures · PDB IDs listed, including 1QYS, 6CZJ, 6D0T, 6MRS, 6NUK, 6W3W, 6WVS
  • ESM2 training corpus after removing sequences labelled artificial construct (1,027) and all jackhmmer hits to the de novo targets (58,462) · natural proteins across evolution
  • Nonredundant PDB set used to train the structure projection, structures released before 1 May 2018 · n = 15,051 proteins
  • Held-out natural proteins with PDB structures, identity < 0.3 to the projection training set, lengths 50 to 250 · n = 214 proteins
  • Unconstrained generations at fixed length 100, structures predicted by the oracle · none (generated sequences) · n = 25,000 sequences
  • Proteins expressed and run on size exclusion chromatography over two rounds: language-model designs, no-language-model baselines and de novo ground-truth controls · n = 276 unique proteins (228 designs, 40 baselines, 8 controls)
  • Search databases for novelty: UniRef90 and UniRef100 2021_04, AlphaFold DB covering 200M UniProt sequences, UniRef50 2018_03 for n-gram background, BLAST non-redundant
Methods & tools
ESM2_650M, AlphaFold, ProteinMPNN, ESM-IF1, ColabDesign (AfDesign), Rosetta (PackStat, SAP, beta_nov16), jackhmmer (HMMER), Foldseek, TM-align, BLAST, Biotite, BioPython
Reagents & environment
ESM2_650M; jackhmmer 3.3.2 run as -n 1 -seed 0; Foldseek version 7d0c07f89a; ColabDesign commithash e7bb3def; UniRef90 and UniRef100 2021_04; UniRef50 release 2018_03; BLAST non-redundant database downloaded Sept 12, 2022; Rosetta beta_nov16 energy function.
Assay conditions & compute
About 10 hours per fixed backbone design trajectory at length about 100 on a single 32GB Volta GPU, with 170,000 Markov chain Monte Carlo steps per trajectory.
Reproducibility gaps
No repository or dataset link survives in this text (an AlphaFold DB url is cited but stripped); designed sequences are named only by F/G identifiers, and the only identifier given for a physical reagent is the entry vector on addgene (ID: 191551).
Limitations (5)
  • The authors note that only high-resolution structural work could confirm the hydrogen-bond networks central to the generalization claim; the measured behaviour of the designs is offered as indirect support. · stated by the authors
  • The authors caution that generalization need not mean the model has learned a physical energy; it may be learning patterns that only approximate one in the limit of unlimited sequence data. · stated by the authors
  • Homology at the edge of detection is not excluded in general: top jackhmmer hits were non-structural matches for only 19 of 81 fixed backbone successes, and a likely different fold for 8 designs spanning 4 backbones. · stated by the authors
  • Success means soluble with a size exclusion peak at the expected elution volume; no design had its structure solved experimentally, so every structural claim rests on AlphaFold predictions. · noted in this summary
  • Apart from the LM versus no-LM round, tested designs passed oracle RMSD, pTM, SAP, packing and globularity filters and in places manual inspection, so headline success rates describe curated candidates, not raw samples. · noted in this summary

Read this if you work on generative protein design or on what sequence-trained models learn, and want wet-lab evidence, with a matched no-language-model control, on designs far from natural sequence space.

Citations
181

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISFerruz '22Madani '21Hie '22Zheng '23Madani '20Akpinaroglu '23
85% semantic matchBioinformatics
Noelia Ferruz, Steffen Schmidt, Birte Höcker·Nature Communications·Jul 27, 2022· 797 citations