Richard Socher

This paper shows that deep learning language models can generate functional artificial proteins across families, with activities similar to natural proteins and structures resembling natural folds.

One attention head of a protein BERT model tracks 3D contacts (28% vs 1.3% background); other heads target binding sites, and deeper layers attend to higher-level properties.

A 1.2B-parameter autoregressive model trained on 280M tagged sequences completes held-out proteins with near-native Rosetta energies and ranks high-fitness GB1 variants without any GB1 training.