Caiming Xiong

This paper shows that deep learning language models can generate functional artificial proteins across families, with activities similar to natural proteins and structures resembling natural folds.

One attention head of a protein BERT model tracks 3D contacts (28% vs 1.3% background); other heads target binding sites, and deeper layers attend to higher-level properties.