Language models enable zero-shot prediction of the effects of mutations on protein function

Key results

1. 01

   ShownZero-shot ESM-1v performs on par with the MSA-trained DeepSequence model and better than earlier protein language models at predicting measured mutational effects.

   Across 41 deep mutational scans scored by Spearman correlation, ESM-1v correlates better with experiment than DeepSequence on 17 of 41 datasets, and a paired t-test cannot separate the two; ESM-1v outperforms TAPE, UniRep, ProtBERT-BFD and ESM-1b zero-shot (per-model values sit in tables not included in this text). · 4.2 Language models enable zero-shot and…

2. 02

   ShownUnsupervised fine-tuning on the family MSA helps only when regularised by mixing pre-training sequences into each batch ('spiked' fine-tuning); naive fine-tuning overfits.

   Spiked fine-tuning (MSA-to-pre-training token ratio 0.01) trains all parameters without overfitting and lifts average absolute Spearman rho from 0.510 zero-shot to 0.537 on the full set; tuning only embeddings or layer norms gave no gain, and minimal PSSM-projection models moved Spearman rho by at most 0.01. · 4.4 Unsupervised fine-tuning on MSAs; Unsupervised fine-tuning baselines; Minimal models; Spiked unsupervised fine-tuning

3. 03

   ShownInference choices matter: masked-marginal scoring is best, a single wildtype forward pass is nearly as good, and MSA Transformer few-shot predictions improve with more diverse context sequences.

   Masked marginals beat wildtype marginals, mutant marginals and pseudolikelihood on the validation set; the one-pass wildtype marginal lost only about 1% absolute performance. For MSA Transformer, sequence-weighted sampling beat random sampling, which beat diversity-maximising and then diversity-minimising selection, with the best results at 384 sequences. · Scoring methods; 4.3 MSA Transformer; Wildtype marginal probability (1 forward pass)

4. 04

   ShownPre-training data clustering and model size both shape zero-shot accuracy: UniRef clustered at 90% identity works best, UniRef100 deteriorates early in training despite being the largest set, and larger models score higher.

   Compared with the 50% identity clustering used for ESM-1b, 70% clusters improved results and 90% gave the largest gain; UniRef100, though largest, deteriorated early in training. Performance rose with parameter count. ESM-1v is a 650M-parameter model trained on 98 million UniRef90 2020-03 sequences (held-out perplexity 7.29). · ESM-1v; Pre-training data; Parameter count

5. 05

   ShownWithout supervision, ESM-1v's per-position uncertainty reflects structure and function: low-entropy positions concentrate at binding sites and in the protein core, and its amino-acid probabilities are well calibrated except for methionine.

   Entropy differs significantly between binding and non-binding residues and is significantly lower for buried than surface sites; the 10 lowest-entropy residues of M.HaeIII sit in the active site. On 15008 trRosetta sequences, average calibration error is 0.006 excluding the first residue; model entropy correlates with PSSM entropy at Pearson r = 0.44. · Protein structure and function; Calibration