Transformer protein language models are unsupervised structure learners

Key results

1. ShownWith a sparse logistic regression on attention maps trained on 20 proteins, ESM-1b predicts short-, medium- and long-range contacts more precisely than Gremlin, both with matched sequence data and, on average, when Gremlin gets optimised trRosetta MSAs that include metagenomic sequences.

   Test set of 14842 trRosetta proteins (20 used for training, 20 for validation); confidence intervals within 0.5 percentage points. Gremlin ran on Jackhmmer MSAs built from the ESM training set (a UniRef50 subset) or on trRosetta MSAs from UniRef100 plus metagenomics. The per-range precision table is not in this text; on 15 CASP13 free-modelling domains ESM-1b also beats Gremlin. · 5 Results; 1 Introduction; 4.2 Gremlin; A.6 Results on CASP13

2. ShownContacts are learned in unsupervised pre-training, not by the supervised read-out: a single training protein, a few top heads with the regression weights discarded, or Gremlin-derived pseudo-labels instead of structures already roughly match or beat Gremlin.

   One training protein: long-range top-L precision 39.2, not distinguishable from Gremlin (p > 0.05); 10 beat it on all metrics. Over 100 bootstrapped fits, mean 35.6 (SD 8.9) with 1 protein, 40.1 (SD 0.3) with 20. Top-5 heads averaged beat Gremlin on matched data; MSA-derived labels give long-range P@L 39.9. A bilinear model on 700x more structures: 20.1 vs 18.6 for the regression. · Low-N; Top Heads; MSA Only; 5 Results

3. ShownThe advantage is largest for short- and medium-range contacts and for proteins with shallow alignments; at long range, against Gremlin with metagenome-augmented MSAs, ESM-1b is only slightly better and wins on just over half of proteins.

   ESM-1b has higher long-range P@L than Gremlin (trRosetta MSAs) on 55% of test proteins and is better at short and medium range at every MSA depth. At long range it wins for MSA depth < 1000 and also > 16000, where Gremlin is known to plateau. With matched sequence data it beats Gremlin at all ranges regardless of depth. · 5.3 Performance Distribution; A.7 Performance Distribution

4. SuggestedLong-range contact precision is linearly related to masked-language-model validation perplexity within each model family, so perplexity can serve as a proxy for how much structure a model has learned, and both scale and hyperparameters matter.

   At equal perplexity the 12-layer ESM-1 matches the long-range P@L of the 34-layer ESM-1. ESM-1 and ESM-1b (similar parameter counts, different hyperparameters and architecture) each show a linear trend but are not directly comparable because they use different masking. The plotted values are not in this text. · 5.5 Perplexity vs. Contact Precision

5. ShownThe regression's contact probabilities are well calibrated, most false positives sit next to true contacts, and ESM-1b tolerates long alanine insertions, consistent with an implicit alignment of the sequence.

   Calibration mean-squared error 0.014 for ESM-1b vs 0.074, 0.028 and 0.020 for 6-, 12- and 34-layer ESM-1. False positives are mostly within Manhattan distance 1-4 of a true contact; in one homodimer example, distant ones matched inter-chain contacts. On 1000 sequences, precision holds up to 256 alanines inserted at either end and 64 in the middle; TAPE lost 12 points after 8 alanines on 1a27. · 5.6 Calibration, False Positives, and Robustness; A.11 Model Calibration and False Positives; A.12 Alignment