henhao
Sign in
BioinformaticsbioRxiv
Preprint

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Jesse VigiD, Ali MadaniiD, Lav R. VarshneyiD, Caiming XiongiD, Richard SocheriD, Nazneen Fatema RajaniiD

PDFDOICode
Post to X
Threads on XPost to BlueskyThreads on Bluesky
In-depth full-text digest available · Jump to digest ↓

Abstract

Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. Through the lens of attention, we analyze the inner workings of the Transformer and explore how the model discerns structural and functional properties of proteins. We show that attention (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We also present a three-dimensional visualization of the interaction between attention and protein structure. Our findings align with known biological processes and provide a tool to aid discovery in protein engineering and synthetic biology. The code for visualization and analysis is available at https://github.com/salesforce/provis.

02 /Specification & Classification

Click any tag to browse matching papers
Citations
338
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (13 Jul 2020) · CC BY-NC-ND 4.0

One attention head of a protein BERT model tracks 3D contacts (28% vs 1.3% background); other heads target binding sites, and deeper layers attend to higher-level properties.

Transformer protein language models learn useful representations but are opaque, and one earlier study probed their output embeddings without examining attention. Adapting NLP 'BERTology' methods, the authors measure how attention lines up with residue contacts, binding sites and amino acid substitution relationships, and test whether attention weights are calibrated, which they say had not been examined before.

Key results

  1. ShownA single head (12-4) out of 144 concentrates on residue pairs that are in 3D contact, although the model was trained on sequence alone with no structural inputs or labels.

    28% of head 12-4's attention falls on contact pairs, against at most 7% for any other head and a 1.3% contact frequency among all token pairs; when its attention weight exceeds 0.9, alignment rises to 76%. Contacts are residues within 8 angstroms of each other but at least 6 positions apart in sequence. · Attention aligns strongly with contact maps in…; Secondary and tertiary structure

  2. ShownIn most layers attention targets binding sites above their background frequency, most strongly in the last six layers, and it often reaches them from distant sequence positions.

    Binding sites make up 4.8% of residues; in the last 6 layers, 15 heads each put over 20% of their attention on them, and head 7-1 puts 34%. Attention from head 7-1 to binding sites spans 124 tokens on average. Per-layer means are in a figure not included in this text. · Attention targets binding sites, especially in…; Probing tasks

  3. ShownHead 12-4's attention weight tracks the empirical chance that two residues touch, which the authors read as attention acting as a calibrated confidence for contacts.

    Residue pairs were binned by head 12-4 attention weight and the share of pairs in contact was computed per bin; the Pearson correlation between bin midpoints and those contact proportions is 0.97. · Attention is a well-calibrated predictor of…

  4. ShownDeeper layers shift attention toward binding sites and contacts while secondary structure is attended more evenly across layers; linear probes likewise find secondary structure encoded before contacts and binding sites.

    Per-layer attention and probe curves (F1 for secondary structure, precision@L/5 for contacts, precision@L/20 for binding sites) are in figures and an appendix not included in this text. Contact probing drops at the final layer, so the authors suggest the second-to-last layer for pairwise downstream tasks. · Attention targets higher-level properties in…; Probing tasks

  5. ShownIndividual heads specialise in particular amino acids, and amino acids that receive similar attention across heads tend to be ones the BLOSUM matrix scores as interchangeable.

    For 14 of the 20 amino acid types some head puts over 25% of its attention on that type (head 1-11: 78% on Pro; head 12-3: 27% on Phe), though no amino acid exceeds 9.4% of residues. Pairwise attention-profile correlations versus BLOSUM scores give a Pearson correlation of 0.80, over 5000 ProteinNet sequences. · Attention heads specialize in certain types of…; Attention is consistent with substitution…

Methods, data and reproducibility

For all 144 heads of the TAPE BERT-Base model (masked language modeling on 31 million protein sequences), the share of attention landing on residue contacts, binding sites, secondary structure classes or specific amino acids was compared with background frequencies; linear probes on the frozen model tested what each layer encodes, and a 3D viewer overlays attention on structures.

  • ProteinNet from the TAPE repository, with residue coordinates used to build contact maps; used for amino acid and contact analyses · n = 5000 sequences (1,067,712 amino acids) for attention analysis
  • TAPE Secondary Structure dataset, used for secondary structure analyses · n = 5000 training sequences for attention; full splits for probing
  • Binding-site set: Secondary Structure sequences with token-level binding-site labels from the Protein Data Bank Web API; unlabelled sequences dropped · n = 5000 sequences for attention; full size in a table not included
  • Pretraining corpus of the TAPE BERT-Base model · n = 31 million protein sequences
Methods & tools
BERT-Base protein model (TAPE), TAPE, ProteinNet, Protein Data Bank Web API, BLOSUM
Assay conditions & compute
A single Tesla V-100 GPU with 16GB memory; sequences truncated to 512 tokens to limit memory use.
Reproducibility gaps
Visualization and analysis code is said to be available, but the link was lost in extraction. Datasets come from the TAPE repository and the Protein Data Bank with no links here, and the detailed probing appendix is empty in this text.
Limitations (5)
  • The analysis shows associations between attention and protein properties only; it does not establish that attention drives model behaviour, nor does it explain predictions. · stated by the authors
  • Head 12-4 correlates strongly with classically defined contacts but also departs from them, possibly reflecting a different learned notion of residue interaction. · stated by the authors
  • Protein databases used to train such models may be skewed toward a few model organisms, and that bias could carry into the models and their interpretation. · stated by the authors
  • Only one model, the TAPE BERT-Base, is analysed; whether the same heads and layer trends appear in other protein Transformers is not tested. · noted in this summary
  • Binding-site attention is described as significantly above background, but no statistical test is reported in this text. · noted in this summary

Who should read:Read this if you use protein language model embeddings or attention maps and want evidence on which heads and layers carry contact, binding-site and substitution information.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISLi '24Bhattachary '21Carbone '22Costa '21Nambiar '20Brandes '22
65% semantic matchBioinformatics
Mingchen Li, Yang Tan, Xinzhu Ma+5 more·neural information processing systems·Jan 1, 2024· 47 citations