bioRxivSynthetic BiologyPreprint
MSA Transformer
Roshan RaoiD, Jason J. LiuiD, Robert VerkuiliD, Joshua MeieriD, John CannyiD, Pieter AbbeeliD, Tom SercuiD, Alexander W. RivesiD
Abstract
Unsupervised protein language models trained across millions of diverse sequences learn structure and function of proteins. Protein language models studied to date have been trained to perform inference from individual sequences. The longstanding approach in computational biology has been to make inferences from a family of evo lutionarily related sequences by fitting a model to each family independently. In this work we combine the two paradigms. We introduce a protein language model which takes as input a set of sequences in the form of a multiple sequence alignment. The model interleaves row and column attention across the input sequences and is trained with a variant of the masked language modeling objective across many protein families. The performance of the model surpasses current state-of-the-art unsupervised structure learning methods by a wide margin, with far greater parameter efficiency than prior state-of-the-art protein language models.
Abstract summary
Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the abstract · CC BY-NC-ND 4.0
One protein language model trained across many families on whole multiple sequence alignments, with interleaved row and column attention, reportedly outperforms unsupervised structure-learning methods by a wide margin.
Protein language models have so far made predictions from single sequences, while the classic approach fits a separate model to each family of related sequences. This model merges the two: it is trained once across many families but takes a whole multiple sequence alignment (MSA) as its input.
Key results
ClaimedThe MSA-input language model is reported to beat current leading unsupervised methods for learning protein structure by a wide margin.
Stated in the abstract without metrics, benchmark datasets or comparison values; the specific structure task and the methods compared are not named in this text. · Abstract
ClaimedIt is reported to be far more parameter-efficient than earlier leading protein language models that take single sequences.
Stated in the abstract; parameter counts and the models compared are not given in this text. · Abstract
Methods, data and reproducibility
A protein language model whose input is a multiple sequence alignment; attention alternates between the rows and the columns of the alignment, and training uses a modified masked-language-modeling objective over many protein families.
- Multiple sequence alignments spanning many protein families (training data; source not named in this text)
- Methods & tools
- MSA Transformer
- Reproducibility gaps
- Abstract only: no code, model weights, training data sources, parameter counts or compute details are given in this text.
Limitations (2)
- The abstract gives no numbers, so the size of the margin over unsupervised structure-learning methods, the benchmarks used and the parameter counts behind the efficiency claim cannot be checked from this text. · noted in this summary
- The model requires a multiple sequence alignment as input; how it behaves for proteins with few or shallow alignments is not addressed in the abstract. · noted in this summary
Read this if you build or use protein language models or alignment-based structure methods and want a single model that learns from whole multiple sequence alignments across protein families.
- Citations
- 350
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Generative power of a protein language model trained on multiple sequence alignments
Damiano Sgarbossa, Umberto Lupo, Anne‐Florence Bitbol·eLifePeer-reviewed·Feb 3, 2023
- Transformer protein language models are unsupervised structure learners
Roshan Rao, Joshua Meier, … Alexander W. Rives·bioRxiv·Dec 15, 2020
- End-to-end multitask learning, from protein language to protein features without alignments
Ahmed Elnaggar, Michael Heinzinger, … Burkhard Rost·bioRxiv·Jan 24, 2020
- Scaling down protein language modeling with MSA Pairformer
Yo Akiyama, Zhidian Zhang, … Sergey Ovchinnikov·bioRxiv·Aug 2, 2025
- Protein language models trained on multiple sequence alignments learn phylogenetic relationships
Umberto Lupo, Damiano Sgarbossa, Anne‐Florence Bitbol·Nature CommunicationsPeer-reviewed·Oct 22, 2022
- xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
Bo Chen, Xingyi Cheng, … Le Song·bioRxiv·Jul 5, 2023