henhaobio
Sign in

neural information processing systemsSynthetic BiologyPeer-reviewed journal article

Scaling unlocks broader generation and deeper functional understanding of proteins

Aadyot Bhatnagar, Sarthak Jain, Joel Beazer, Samuel C. Curran, Alexander M. HoffnagleiD, Kyle Ching, Michael Martyn, Stephen NayfachiD, Jeffrey A. RuffoloiD, Ali MadaniiD

PDFDOIPreprint
In-depth full-text digest available · Jump to digest ↓

Abstract

Generative protein language models (PLMs) are powerful tools for designing proteins purpose-built to solve problems in medicine, agriculture, and industrial processes. Recent work has trained ever larger language models, but there has been little systematic study of the optimal training distributions and the influence of model scale on the sequences generated by PLMs. We introduce the ProGen3 family of sparse generative PLMs, and we develop compute-optimal scaling laws to scale up to a 46B-parameter model pre-trained on 1.5T amino acid tokens. Pro-Gen3s pre-training data is sampled from an optimized data distribution over the Profluent Protein Atlas v1, a carefully curated dataset of 3.4B full-length proteins. We evaluate for the first time in the wet lab the influence of model scale on the sequences generated by PLMs, and we find that larger models generate viable proteins for a much wider diversity of protein families. Finally, we find both computationally and experimentally that larger models are more responsive to alignment with laboratory data, resulting in improved protein fitness prediction and sequence generation capabilities. These results indicate that larger PLMs like ProGen3-46B trained on larger, well-curated datasets are powerful foundation models that push the frontier of protein design.1

In-depth summary

Claude Opus 5.5 · xhigh effort · 5 Nov 2026 · from the full text of version 2 (1 Oct 2025) · CC BY-NC-ND 4.0

Scaling sparse Mixture-of-Experts protein language models up to 46B parameters on 1.5T curated tokens broadens viable protein generation across families and amplifies gains from laboratory preference alignment.

Prior protein language model studies relied on ad-hoc training data distributions and rarely tested how model scale affects wet-lab viability or post-training alignment. Deriving compute-optimal scaling laws and cluster-rebalanced sampling on 3.4B full-length proteins enables efficient sparse models that generate soluble proteins across rarer families.

Key results

  1. ShownInverse Log cluster sampling optimizes out-of-distribution generalization and supports compute-optimal scaling up to a 46B-parameter sparse model.

    Among 1.4B-parameter models trained on 80B tokens from the Profluent Protein Atlas v1 (3.4B proteins, 1.1T tokens), Inverse Log sampling over 50% and 90% identity clusters yielded the lowest validation loss at 50% and 30% identity, while ProGen3-46B trained on 1.5T tokens (1.1 x 10^23 FLOPs) reached a validation loss of 1.345 vs the predicted 1.397. · 2.2 Training Data; 2.3 Scaling Up to a 46B Parameter Model; B.4 Importance of Warmup Duration; Figure legends

  2. ShownLarger ProGen3 models generate higher proportions of valid sequences covering substantially more diverse protein families that express solubly in vitro at natural-like rates.

    Unconditional generation yielded 30%, 40%, and 49% valid sequences for ProGen3-339M, 3B, and 46B vs 4% for dense 6.4B ProGen2-XL; ProGen3-3B covered 81% more 30% identity clusters than 339M, and in split-GFP E. coli assays across 42 shared, 62 3B/46B-shared, and 45 46B-only clusters, all models matched natural control expression rates. · 3.1 Larger PLMs generate more diverse proteins…; Figure legends

  3. ShownProGen3 generates soluble proteins outside known natural sequence clusters and infills internal sequence spans more effectively at larger scale.

    Across 90 unconditional generations with <30% sequence identity to any PPA-1 protein (10 each of mostly-alpha, mostly-beta, and mixed structure per model), larger models achieved higher split-GFP expression levels, and ProGen3-46B outperformed ProGen3-3B when infilling 20% and 50% length spans across 18 proteins. · 3.2 PLMs generate viable proteins outside…; 3.3 Larger PLMs are better infillers; Figure legends

  4. ShownSupervised alignment with iterative reasoning preference optimization (IRPO) reverses high-scale zero-shot degradation and improves distant mutation and stability prediction.

    While zero-shot ProteinGym performance dropped above 3B parameters, IRPO-aligned ProGen3-46B achieved mean Spearman rho = 0.673 across 8 ProteinGym assays (vs 0.628 for KERMUT and 0.679 for ConFit) and rho = 0.737 overall (rho = 0.820 on multi-substitutions) on Megascale stability vs 0.72 (0.468 on multi-substitutions) for ProteinDPO. · 4 Aligning Protein Language Models to Laboratory…; 4.1 Supervised Fitness Prediction; C.3 ProteinGym Alignment Experiments; Figure legends

  5. ShownAligning ProGen3 on folding stability data improves both Rosetta energy and experimental soluble expression of conditionally generated proteins.

    Across 32 natural protein prompts (98-282 amino acids), stability alignment lowered PyRosetta minimum energy across scales and significantly increased in vitro split-GFP expression for 8/32 prompts with ProGen3-46B (degrading 3/32 and leaving 21/32 unchanged; paired Welch's t-test, p < 0.01). · 4.2 Sequence Generation; Figure legends

Methods, data and reproducibility

Curated the Profluent Protein Atlas v1 (PPA-1) by filtering 12.2B proteins to 3.4B full-length sequences clustered at 90%, 50%, and 30% identity with DIAMOND. Trained sparse Mixture-of-Experts transformers (ProGen3, 112M to 46B parameters, activating 2 of 8 experts) on span-infilling and causal language modeling, aligned models to DMS data via IRPO, and tested proteins with ESMFold, PyRosetta, and E. coli split-GFP.

  • Profluent Protein Atlas v1 (PPA-1) full-length proteins from IMG/M, ENA, NCBI GenBank, UniRef, and NCBI NR · n = 3.4B proteins, 1.1T tokens (from 12.2B raw proteins; 113M 30% ID clusters)
  • ProteinGym deep mutational scanning assays for supervised IRPO fitness alignment · n = 8 assays (>500 training sequences at most k mutations from wild type)
  • Megascale protein folding free energy (ΔG) dataset of single- and double-mutation variants · n = 800,000 variants across 479 protein domains
  • In-lysate E. coli BL-21 DE3 split-GFP soluble expression assays of unconditional, <30% ID, infilled, and stability-aligned generations · E. coli · n = 42, 62, 45 clusters; 90 <30% ID proteins; 18 infill targets; 32 stability cases · NP_000782.1
Methods & tools
ProGen3, ProGen2-XL, DIAMOND, prodigal-gv, PyTorch, FlashAttention2, Megablocks, ESMFold, PyRosetta, FoldSeek, minimap2, samtools
Reagents & environment
PyTorch with BF16 mixed precision, FlashAttention2, Megablocks, MosaicML composer and streaming; sequence verification with Jellyfish (kmer 31), Uniquekmer, BBDuk, BBSeal, minimap2, samtools, and DNADiff.
Experimental conditions
ProGen3-46B pre-trained for ~17 days on 256xH100 GPUs (1.1 x 10^23 FLOPs) on Databricks and runs inference on 4xA100_40GB GPUs; IRPO alignment took <3hr on 8xH100 (<46B) or 2-6hr on 16xH100 (46B).
Reproducibility gaps
Model code and weights up to ProGen3-3B are stated as released on GitHub and Huggingface without URLs in the text, whereas PPA-1 data, training code, and ProGen3-46B weights are not released.
Limitations (4)
  • The split-GFP E. coli assay measures soluble protein abundance rather than directly testing catalytic, binding, or cellular functions of diverse protein families. · stated by the authors
  • Computational pre-training and IRPO alignment experiments used single runs and a single train/validation/test split without cross-validation or error bars due to computational cost. · stated by the authors
  • Both aligned ProGen3 models and baselines performed worst on ProteinGym binding affinity assays (GCN4_YEAST_Staller_2018 and SPG1_STRSG_Wu_2016), as sequence-only pre-training lacks binding partner information. · stated by the authors
  • The curated PPA-1 dataset, pre-training code, and flagship ProGen3-46B weights are closed-source and unavailable for external verification. · stated by the authors

Read this if you train or align protein language models and want scaling laws, data-balancing strategies, or preference-optimization methods for sequence generation and fitness prediction.

Citations
44
FWCI
169.78
Field percentile
top 1%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISFournier '24Garibbo '26Chen '23Verkuil '22Serrano '23Madani '20
80% semantic matchMolecular Biology
Quentin Fournier, Robert McCoy Vernon, Almer M. van der Sloot+3 more·bioRxiv·Sep 23, 2024· 27 citations