Roshan Rao

The paper uses language modeling to create a comprehensive representation of protein biology, enabling structure prediction and discovery of high-affinity protein interactions.

ESM3, a 98-billion-parameter multimodal protein language model tokenized across sequence, structure, and function, designs a bright green fluorescent protein at 58% sequence identity to known fluorescent proteins.

Protein language models scaled to 15B parameters predict atomic-resolution structures from sequence up to 60x faster than the state of the art, producing over 617 million metagenomic structures.

This paper develops machine learning models to predict antibody thermostability using sequence and structure features, showing better generalization with a CNN over pre-trained language models.

A modular high-level programming language paired with an energy-based protein language model generates de novo all-atom protein sequences and structures satisfying user-specified geometric, secondary structure, symmetry, and multimerization constraints.

This paper introduces a differentiable Smith-Waterman algorithm to jointly learn multiple sequence alignments and downstream tasks, showing improvements in contact prediction and structure prediction.

The paper shows that a single attention layer can predict protein contacts as well as Potts models and pretrained Transformers, suggesting attention models can capture meaningful protein features from unlabeled data.

A single pre-trained protein language model, ESM-1v, scores mutations zero-shot about as well as family-specific DeepSequence across 41 deep mutational scans, with no experimental labels or retraining.

One protein language model trained across many families on whole multiple sequence alignments, with interleaved row and column attention, reportedly outperforms unsupervised structure-learning methods by a wide margin.

Attention maps of the 650M-parameter protein language model ESM-1b, read out by a logistic regression fitted on 20 proteins, predict contacts better on average than Gremlin's alignment-based Potts models.

Across five protein tasks with evolution-aware splits, self-supervised pretraining helped nearly every model on every task, yet alignment-based features still beat learned embeddings on secondary structure and contact prediction.