henhaobio
Sign in

bioRxivSynthetic BiologyPreprint

ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction

Pascal NotiniD, Aaron W. KollaschiD, Daniel RitteriD, Lood van NiekerkiD, Steffan B. PauliD, Hansen Spinner, Nathan J. RollinsiD, Ada Y. ShawiD, Ruben WeitzmaniD, Jonathan FrazeriD, Mafalda DiasiD, Dinko Franceschi, Rose OrenbuchiD, Yarin GaliD, Debora S. MarksiD

PDFDOICodeData

In-depth summary

Claude Sonnet 4.6 · max effort · 8 Nov 2026 · from the full text of version 1 (8 Dec 2023) · CC BY-NC-ND 4.0

ProteinGym unifies over 250 deep mutational scans and ~65k clinical variants to benchmark over 70 models in zero-shot and supervised settings; TranceptEVE ranks best overall.

Earlier protein model evaluations relied on distinct, sparse datasets, and model rankings shift across assays. Existing multi-task benchmarks include only 1-3 fitness assays, so ProteinGym adds two orders of magnitude more point mutations plus clinical labels, zero-shot and supervised regimes, and design-oriented metrics.

Key results

  1. ShownTranceptEVE is the best overall method across the settings reported and tops most metrics. GEMME leads in some subgroups: viral and non-human eukaryotic assays, and low and medium MSA depths.

    Rankings from Table 2 (average Spearman, AUC, MCC, NDCG@10%, top-10% recall) and breakdowns by MSA depth, mutational depth and taxa. Table A5 gives bootstrap standard errors (10k samples) versus TranceptEVE. Numeric table values are not in the text. · 5.1 Substitution benchmarks; Zero-shot; Tables

  2. SuggestedModels rank differently under Spearman and NDCG. Alignment-based models (EVmutation, WaveNet, DeepSequence) tend to rank better on NDCG, which may suit design. ESM-1v and ESM-2 tend to rank better on Spearman. Autoregressive language models tend to beat masked ones zero-shot.

    Median NDCG@10% versus median Spearman across zero-shot models on DMS substitutions (Fig. 2B); hierarchical clustering of NDCG vectors groups alignment-based models together. No numeric values are given in the text. · 5.1 Substitution benchmarks; Clustering zero-shot substitution models; Figure legends

  3. ShownIn the supervised DMS setting, ProteinNPT achieves the best performance. Tests use 5-fold Random, Contiguous and Modulo cross-validation, so models are scored on extrapolation to unseen positions.

    Table 3 (Spearman and MSE) against one-hot-encoding and embedding baselines augmented with zero-shot predictions; Table A10 covers 10 baseline models. Numeric values are not given in the text. · DMS assays; 5.1 Substitution benchmarks; Supervised

  4. ShownOn clinical labels, most supervised variant predictors are outperformed by the best unsupervised ones, even though supervised models may benefit from leakage. TranceptEVE outperforms supervised ClinPred on the DMS subset of human clinical proteins.

    Fig. 2A plots average per-gene AUC on ClinVar against average Spearman on human clinical DMS assays. The clinical set has 2,525 proteins and 63k variants. Numeric scores are not in the text. · Clinical datasets; ClinVar processing; A.5.3 Clinical substitution benchmarks

  5. ShownFor indels, Tranception Medium has the highest Spearman and AUC across all assays. TranceptEVE is best on the subset of designed or natural-biased sequence libraries.

    Table 4, an indel-compatible subset of models (Tranception, TranceptEVE, WaveNet, HMM, ProGen2, UniRep, RITA, ProtGPT2), Spearman split by unbiased libraries versus designed sequences. Numeric values are not in the text. · Baselines; 5.2 Indel benchmarks; Zero-shot

Methods, data and reproducibility

Curated substitution and indel DMS assays plus ClinVar/gnomAD clinical sets, then scored over 70 zero-shot and supervised models in one codebase. DMS metrics: Spearman, AUC, MCC, NDCG, top-K recall. Clinical metrics: areas under ROC and precision-recall curves.

  • Standardized DMS assays (substitutions and indels) across more than 200 protein families, with over 2.7 million mutated sequences · Multiple (human, other eukaryotes, prokaryotes, viruses) · n = over 250 assays
  • Thermostability assays derived from a large dataset of natural domains (non-redundant set, 40-72 residues) · n = 65 assays from 331 domains
  • Clinical substitution benchmark (ClinVar, EVE pipeline remapped to GRCh38) · human · n = 2,525 proteins; 63k variants
  • ClinVar inframe indels after filtering (up to 3 amino acids); 330 benign, 1760 pathogenic · human · n = 2090 of 18407 variants
  • gnomAD v2.1.1 GRCh38 liftover common indels used as benign pseudocontrols · human · n = 839 indels (125,748 exomes; 15,708 genomes)
  • Cross-validation of supervised DMS tasks (Random, Contiguous, Modulo) · n = 5 folds
Methods & tools
TranceptEVE, Tranception, GEMME, EVE, EVmutation, DeepSequence, WaveNet, profile HMM (HMMER), ESM-1v / ESM-2 / ESM-IF1, ProteinMPNN / MIF, MSA Transformer, ProteinNPT
Software & environment
Codebase is MIT licensed. Clinical inputs: gnomAD v2.1.1 GRCh38 liftover (downloaded February 8th 2023), ClinVar indels (queried February 6th, 2023), dbNSFP v4.4a.
Reproducibility gaps
Most model checkpoints (e.g. ESM-1v) must be obtained from their original repositories. GEMME scores came from an external web tool.
Limitations (5)
  • DMS assays have measurement noise, limited dynamic range, biased protein selection (viral and cancer proteins over-represented), and single-feature readouts. Heterogeneous data processing across sources is not standardized. · stated by the authors
  • ClinVar labels are noisy, biased toward well-studied genes and European ancestry, and circular (conservation informs benign/pathogenic calls). Supervised models can leak training labels, and benign labels reflect population frequency. · stated by the authors
  • Cross-gene calibration of clinical scores is unresolved, and the benchmarks cover only coding-region mutations. · stated by the authors
  • Hybrid cross-family generalization (training on a few proteins, testing on unseen families) is not evaluated. · stated by the authors
  • The text gives gnomAD common-indel pseudocontrol thresholds inconsistently (allele frequency >5% in one section, >0.5% in another), leaving the clinical indel control definition ambiguous. · noted in this summary

Read this if you need a standardized benchmark to compare zero-shot or supervised protein fitness and variant effect predictors across DMS and clinical datasets.

Citations
207

Abstract

Predicting the effects of mutations in proteins is critical to many applications, from understanding genetic disease to designing novel proteins that can address our most pressing challenges in climate, agriculture and healthcare. Despite a surge in machine learning-based protein models to tackle these questions, an assessment of their respective benefits is challenging due to the use of distinct, often contrived, experimental datasets, and the variable performance of models across different protein families. Addressing these challenges requires scale. To that end we introduce ProteinGym, a large-scale and holistic set of benchmarks specifically designed for protein fitness prediction and design. It encompasses both a broad collection of over 250 standardized deep mutational scanning assays, spanning millions of mutated sequences, as well as curated clinical datasets providing high-quality expert annotations about mutation effects. We devise a robust evaluation framework that combines metrics for both fitness prediction and design, factors in known limitations of the underlying experimental methods, and covers both zero-shot and supervised settings. We report the performance of a diverse set of over 70 high-performing models from various subfields (eg., alignment-based, inverse folding) into a unified benchmark suite. We open source the corresponding codebase, datasets, MSAs, structures, model predictions and develop a user-friendly website that facilitates data access and analysis.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISSchmirler '26Xu '24Chen '26Fernández-d '20Sruthi '17Woolley '26
72% semantic matchBioinformatics
Robert Schmirler, Maurice Brenner, Sebastian Franz+2 more·bioRxiv·Jun 11, 2026