ProteinAligner: A Multi-modal Pretraining Framework for Protein Foundation Models

Abstract

Protein foundation models, particularly protein language models, have demonstrated strong success in learning meaningful representations of proteins using transformer architectures pretrained on large-scale protein datasets with self-supervised learning. These representations have been highly effective for downstream tasks such as predicting protein functions and properties. However, most current protein foundation models focus on pretraining with amino acid sequences, often neglecting additional modalities like protein structures and related literature, both of which provide valuable insights. To address this gap, we propose a multi-modal pretraining approach that integrates three key modalities - protein sequences, structures, and literature text. In our framework, the protein sequence modality serves as the anchor, with the other two modalities aligned to it, enhancing the models capacity to capture more comprehensive protein information. ProteinAligner out-performed state-of-the-art protein foundation models in predicting protein functions and properties across diverse down-stream tasks.