henhao
Sign in
BioengineeringbioRxiv
Preprint

FLIP2: Expanding Protein Fitness Landscape Benchmarks for Real-World Machine Learning Applications

Kieran DidiiD, Sarah AlamdariiD, Alex Xijie LuiD, Bruce J. WittmanniD, Kadina E. JohnstoniD, Ava P. AminiiD, Ali MadaniiD, Maya Czeneszew, Christian DallagoiD, Kevin K. YangiD

Abstract

Machine learning methods that predict protein fitness from sequence remain sensitive to changes in data distributions, limiting generalization across common conditions encountered in protein engineering. Practically, protein engineers are thus left wondering about the effective utility of ML tools. The FLIP benchmark established protocols for testing generalization under some domain shifts, but it was limited to measurements of thermostability, binding, and viral capsid viability. We introduce FLIP2, a protein fitness benchmark spanning seven new datasets, including enzymes, protein-protein interactions, and light-sensitive proteins, as well as splits that measure generalization relevant to real-world protein engineering campaigns. Evaluating a suite of benchmark models across these datasets and splits reveals that simpler models often matched or outperformed fine-tuned protein language models on FLIP2, challenging the utility of existing transfer learning techniques. Provenance for all datasets has been recorded and we redistribute all data CC-BY 4.0 to facilitate continued progress.

PDFDOIData
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
4
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Sonnet 4.6 · xhigh effort · 8 Nov 2026 · from the full text of version 5 (7 Aug 2026) · CC BY 4.0

FLIP2 adds seven protein fitness datasets and 16 splits for enzymes, light-sensitive proteins, and interactions; simple ridge and zero-shot baselines often rival fine-tuned protein language models.

Protein fitness predictors are sensitive to distribution shift, and the earlier FLIP benchmark covered only thermostability, binding, and viral capsid viability. FLIP2 adds enzymatic and interaction datasets plus splits across wild types, positions, and mutations that mimic engineering campaigns.

Key results

  1. ShownA fine-tuned pLM was the top model on only a minority of splits (7 of 16); ridge regression with likelihoods and zero-shot pLM scores each topped more splits than a naive supervised pLM.

    Best model per split: fine-tuned pLM 7/16, ridge regression with sequence and pLM likelihoods 4/16, zero-shot pLM score 4/16, naive supervised pLM 1/16 (Table A18). · 4.3. Fine-Tuned Protein Language Models

  2. ShownPretraining helped fine-tuning on most splits for CARP-640M but less so for ESMC-300M. Supervised fine-tuning of CARP-640M often failed to beat its own zero-shot score, and hurt on all by-position and all Amylase splits.

    Pretraining improved Spearman on 13 out of 16 tasks for CARP-640M vs 10 out of 16 for ESMC-300M; fine-tuned CARP-640M beat zero-shot on 7/16 splits. Five seeds per model. · 4.3. Fine-Tuned Protein Language Models

  3. ShownAlmost every FLIP2 split was harder than a random split of the same size for ridge regression; wild-type, position, and fitness splits were much harder than number and mutation splits, so random splits overstate performance.

    Ridge regression on one-hot sequences, colored FLIP2 splits compared with grey random splits (Figure 3A-G); no single numeric gap is quoted in the text. · 4.2. Linear Models

  4. ShownZero-shot likelihoods rank variants well within one wild-type protein but poorly across proteins or in two-protein landscapes. The best pLM differs by dataset, so no single pLM wins everywhere.

    Predictive for Amylase, IRED, NucB, TrpB; weaker for Hydro, Rhomax, PDZ3. Dayhoff beat other models on Hydro and Rhomax but underperformed on TrpB. SaProt-650M structure tokens gave no consistent gain. · 4.1. Zero-Shot Likelihood Scores

  5. ShownRidge regression needs different amounts of data depending on dataset type: single-wild-type landscapes need more data to surpass the best zero-shot score, while multi-backbone and two-protein landscapes need very few examples.

    Single-wild-type: Amylase, IRED, NucB, TrpB, where adding zero-shot scores helped. Multi-backbone or paired: Hydro, Rhomax, PDZ3, where linear models quickly beat zero-shot (Figure 3A-G). · 4.2. Linear Models

Methods, data and reproducibility

Seven curated datasets with 16 splits across five generalization categories (number, position, mutation, fitness, wild type) were scored with zero-shot pLM likelihoods, ridge regression on one-hot sequences with or without likelihoods, and fine-tuned pLMs, using Spearman and NDCG.

  • Amylase stain-removal activity variants (up to eight mutations; includes single mutants) · Bacillus subtilis · n = 3,706 variants (488 single mutants)
  • IRED imine reductase activity from error-prone PCR library, microfluidic screen · Streptosporangium roseum · n = 17,143 variants
  • NucB nuclease activity at pH 7, binned into four activity classes · Bacillus licheniformis · n = 55,760 variants
  • TrpB growth-based combinatorial landscapes: nine 3-site (~8,000 variants each) and one 4-site (~160,000 variants) · n = ten sub-datasets, 20 positions
  • Hydrophobic core stability, seven randomized core residues in each of three proteins · n = 24,935 variants · UniProt P06241, P01053, P0A9X9
  • Rhomax peak absorption wavelength of microbial rhodopsin variants and chimeras · microbial rhodopsins · n = 884 variants from 75 sequences
  • PDZ3 binding of PDZ3 and CRIPT peptide double mutants; test filtered to non-additive pairs · n = >200,000 double-mutation sets; 579 test pairs
  • Redistributed FLIP2 benchmark data (CC-BY 4.0) hosted on Zenodo · 10.5281/zenodo.18433203
Methods & tools
ESM2-650M, CARP-640M, SaProt-650M (with and without structure tokens), ESMC-300M, Dayhoff-3b-GR-HM-c, Dayhoff-3b-UR90, Ridge regression (one-hot, plus zero-shot scores), LoRA fine-tuning, Spearman correlation, Normalized discounted cumulative gain (NDCG)
Data & accessions
https://doi.org/10.5281/zenodo.18433203
Reagents & environment
Fine-tuning used LoRA rank 32 with scaling 64, five random seeds with validation-based early stopping, and checkpoints ESM2-650M, CARP-640M, SaProt-650M, ESMC-300M, Dayhoff-3b models.
Reproducibility gaps
The benchmark homepage URL is garbled in the text, and no code repository is given. Source datasets come from prior publications.
Limitations (4)
  • Ridge regression cannot generalize to new positions or new scaffolds, and zero-shot likelihoods deteriorate when ranking variants across different proteins or in two-protein landscapes. · stated by the authors
  • The authors note a dual-use concern: stronger generalization in fitness prediction might also be exploited to engineer harmful agents, so use should follow biosafety and biosecurity norms. · stated by the authors
  • The PDZ3 test set is filtered to 579 non-additive pairs and the Rhomax training set uses only the 5 most common wild types, so those splits probe narrow regimes and may not transfer to other campaigns. · noted in this summary
  • The text reports few numeric metrics; most Spearman and NDCG values sit in appendix tables, and fine-tuning hyperparameter search effort is not described, so the comparison with simple baselines is hard to audit. · noted in this summary

Who should read:Read this if you build or benchmark protein fitness predictors and want realistic splits (wild type, position, mutation) and evidence on when pLM fine-tuning fails to beat simple baselines.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISDallago '21Sundar '24Vithayapale '26Groth '23Zhao '24Greenman '23
82% semantic matchBioengineering
C. Dallago, J. Mou, K. E. Johnston+5 more·bioRxiv·Nov 9, 2021