Robert Verkuil

The paper uses language modeling to create a comprehensive representation of protein biology, enabling structure prediction and discovery of high-affinity protein interactions.

ESM3, a 98-billion-parameter multimodal protein language model tokenized across sequence, structure, and function, designs a bright green fluorescent protein at 58% sequence identity to known fluorescent proteins.

Protein language models scaled to 15B parameters predict atomic-resolution structures from sequence up to 60x faster than the state of the art, producing over 617 million metagenomic structures.

A sequence-only protein language model, sampled under a structure-compatibility energy, produced designs of which 152 of 228 tested behaved as soluble monomers, some with no detectable UniRef90 match.

Adding 12 million AlphaFold2-predicted structures to training lifts native sequence recovery on held-out backbones to 51.6%, versus 42.2% for the best experimental-only model; likelihoods score mutation effects zero-shot.

A single pre-trained protein language model, ESM-1v, scores mutations zero-shot about as well as family-specific DeepSequence across 41 deep mutational scans, with no experimental labels or retraining.

The paper introduces a Neural Potts Model that predicts pairwise coupling matrices for protein families using amortized optimization, showing improved performance on low-depth multiple sequence alignments.

One protein language model trained across many families on whole multiple sequence alignments, with interleaved row and column attention, reportedly outperforms unsupervised structure-learning methods by a wide margin.