henhaobio
Sign in

bioRxivSynthetic BiologyPreprint

ProGen: Language Modeling for Protein Generation

Ali MadaniiD, Bryan McCann, Nikhil NaikiD, Nitish Shirish KeskariD, Namrata AnandiD, Raphael R. EguchiiD, Po‐Ssu HuangiD, Richard SocheriD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on [~]280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (13 Mar 2020) · CC BY-NC-ND 4.0

A 1.2B-parameter autoregressive model trained on 280M tagged sequences completes held-out proteins with near-native Rosetta energies and ranks high-fitness GB1 variants without any GB1 training.

Protein engineering still leans on directed evolution and heuristics, and structure-annotated proteins are three orders of magnitude scarcer than raw sequences. Earlier protein language models learned bidirectional representations for classification and earlier generative models used explicit structure or convolutional and variational-autoencoder sequence models, whereas ProGen brings tag-controlled text generation in the style of CTRL to protein sequences at scale.

Key results

  1. ShownAs a language model ProGen far outperforms uniform and empirical amino acid baselines, generalizes to a random held-out set, and still beats the empirical baseline on 20 unseen Pfam families, where fine-tuning from ProGen also beats training from scratch.

    Perplexity on ID-test (1M random sequences) and OOD-test (100k sequences from 20 Pfam families excluded from training); OOD performance is worse than ID. Fine-tuning on OOD-test-80 for 5 epochs improved OOD-test-20 perplexity over the same architecture randomly initialized. Perplexity values are in a table not included in this text; English Wikipedia models reach 10 to 17 for reference. · ProGen generalizes to the full test set and…; ProGen generalizes to unseen protein families; Fine-tuning ProGen on unseen protein families…; 3.1. Data

  2. ShownPredictions improve with more amino acid context and with more conditioning tags, and the authors read many of the model's errors as substitutions that nature tolerates.

    Perplexity falls and per-token hard accuracy rises toward later portions of proteins and as the number of tags grows. Soft accuracy weighted by BLOSUM62 is more than 20% higher than hard accuracy. Per-position and per-tag values are in figures not included in this text. · ProGen performance improves with increased amino…; BLOSUM62 soft accuracy reveals that ProGen…

  3. SuggestedGenerations look more native at higher structural levels than at the sequence level: they approach the 25% random-mutation baseline in sequence similarity, surpass it in predicted secondary structure with 8 or more tags, and relax to near-native Rosetta energies.

    Best sampling was top-k with k = 1 plus a repetition penalty, giving similarity comparable to mutating 50% of unseen residues; 3 or more tags beat the 50% baseline. After threading onto known structures, ProGen samples had energies near or below their relaxed native templates and much closer to native levels than the 50% and 100% mutation and all-alanine baselines. · ProGen achieves higher sequence similarity…; Sequence similarity suggests that ProGen merely…; After threading and relaxation, samples…

  4. SuggestedCompleting the held-out VEGFR2 kinase domain, ProGen produced sequences that diverge from the native slightly more than a 25% random mutation yet have lower Rosetta energies than every baseline at all generation lengths.

    Residues 806 to 1168 of VEGFR2 (PDB 2XIR), with the remaining sequence given as context. Mean identity to the native sequence was 73.1% versus 74% for the 25% mutation baseline. In an example sample, energy was highest at residues lacking secondary structure. · 4.3. Case Study: Completing VEGFR2 kinase domain

  5. SuggestedWith no supervised training on GB1 data and no fine-tuning on related antibody-binding proteins, ranking GB1 variants by model perplexity picked out high-fitness variants, whereas random mutation mostly yields low or zero fitness.

    149,361 of 160,000 possible variants from saturation mutagenesis at four epistatic positions, with fitness combining stability and IgG-Fc binding; the 100 lowest-perplexity variants were selected. Their fitness values are in a figure not included in this text. · 4.4. Case Study: Zero-shot fitness selection for…

Methods, data and reproducibility

A 36-layer, 8-head causal Transformer (1.2B parameters) was trained for next-token prediction on amino acid sequences prefixed with UniProtKB keyword and NCBI taxonomy tags, with sequences also included reversed and untagged. It was scored by perplexity and accuracy, and its generations by sequence similarity, PSIPRED secondary structure and Rosetta energy against random-mutation baselines.

  • Protein sequences from UniParc, UniProtKB, SWISS-PROT, TrEMBL and Pfam with 1100 keyword tags and 100k NCBI taxonomy tags · multiple species · n = 280M training sequences (over 281M in total)
  • Held-out test sets: OOD-test from 20 Pfam families excluded from training, and a random ID-test · n = OOD-test 100k; ID-test 1M
  • VEGFR2 kinase domain, residues 806-1168, held out of training · PDB 2XIR
  • GB1 fitness landscape from NNK/NNS saturation mutagenesis at four positions, fitness measured by mRNA display with next-generation sequencing · n = 149,361 of 160,000 possible variants
  • SWISS-PROT test proteins with RCSB PDB structures, used for Rosetta energy evaluation
Methods & tools
ProGen, TensorFlow, Cloud TPU v3 Pod, Biopython (Needleman-Wunsch alignment), BLOSUM62, PSIPRED, PSI-BLAST (UniRef90), Rosetta RelaxBB, HHblits
Reagents & environment
Implemented in TensorFlow; no versions stated.
Assay conditions & compute
About two weeks of training on 256 cores of a Cloud TPU v3 Pod, global batch size 64, 1M iterations.
Reproducibility gaps
This text contains no statement on releasing code, model weights or the curated training set, and its result tables and figures are missing, so perplexity, accuracy and fitness values cannot be quoted.
Limitations (5)
  • Even at best, generated sequences do not beat the 25% random-mutation baseline on sequence similarity. · stated by the authors
  • Modeling is worse on unseen protein families (OOD-test) than on in-distribution sequences. · stated by the authors
  • Training curves show no overfitting yet, suggesting the model is not at full capacity and larger models or more compute could still help. · stated by the authors
  • Structural and functional quality is judged only by computational proxies (PSIPRED predictions, Rosetta energies); no generated protein was synthesized or tested in the lab. · noted in this summary
  • Rosetta energies are computed after threading generated sequences onto the native backbone, which tests compatibility with a known fold rather than independent folding. · noted in this summary

Read this if you want to generate or score protein sequences with a tag-conditioned autoregressive language model and see how its outputs fare on sequence, secondary-structure and Rosetta-energy proxies.

Citations
223

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISVerkuil '22Madani '21Johnson '21Garibbo '26Ferruz '22Bhatnagar '25
79% semantic matchSynthetic Biology
Robert Verkuil, Ori Kabeli, Yilun Du+7 more·bioRxiv·Dec 21, 2022· 181 citations