henhao
Sign in
Synthetic BiologybioRxiv
Preprint

xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein

Bo CheniD, Xingyi ChengiD, Pan Li, Yangli‐ao GengiD, Jing GongiD, Shen LiiD, Zhilei BeiiD, Xu TaniD, Boyan WangiD, Xin Ying ZengiD, Chi-Ming LiuiD, Aohan ZengiD, Yuxiao DongiD, Jie TangiD, Le SongiD

PDFDOI
Post to X
Threads on XPost to BlueskyThreads on Bluesky
In-depth full-text digest available · Jump to digest ↓

Abstract

Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science. Trained weight for the xTrimoPGLM model, and downstream datasets are available at https://huggingface.co/proteinglm.

02 /Specification & Classification

Click any tag to browse matching papers
Citations
108
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 5 (3 Dec 2024) · CC BY-NC-ND 4.0

A 100-billion-parameter protein language model trained with masked and autoregressive objectives together, topping prior results on 15 of 18 benchmarks and powering a fast structure predictor.

Protein language models are built either as encoders for prediction tasks (ESM) or as decoders for sequence design (PROGEN), forcing a choice of model per task type. xTrimoPGLM trains both objectives in one General Language Model backbone, first 400 billion tokens of masked-token training, then 600 billion mixing the two, and reports that this combination can be scaled to 100 billion parameters and a trillion tokens.

Key results

  1. ShownAcross 18 prediction tasks covering protein structure, developability, interactions and function, the model beat previously reported best results on 15, with parameter-efficient fine-tuning rather than full fine-tuning.

    Comparisons use ESM2-150M and ESM2-15B plus published task results, with LoRA (rank 8, under 1% added trainable parameters) consistently above frozen-embedding MLP probing. Gains over the small ESM2 were largest on harder tasks (contact map, fluorescence, metal binding, stability), small on easy ones such as antibiotic resistance. Per-task scores sit in Supplementary Table 3, absent here. · Introduction; Evaluation on Protein Understanding benchmarks; 1 Protein Understanding Benchmarks and…

  2. ShownA folding head on the model (xT-Fold) outperformed other single-sequence folding tools, but the gap to alignment-based methods persisted on the harder, more out-of-distribution target set.

    TM-score 0.86 on CAMEO (n=194) and 0.70 on CASP15 (n=56), versus 0.85 and 0.65 for ESMFold and 0.80 and 0.60 for OmegaFold; backbone perplexity on these sets was 4.01 and 4.45 against 5.21 and 6.18 for ESM2-3B. MSA and template methods stayed ahead on CASP15 while running roughly 10 to 50 times slower; 4-bit quantisation and FlashAttention let xT-Fold run on one A100, its folding module being about 90M parameters. · Evaluation on Protein Structure Prediction

  3. ShownUnconditional generation produced structures resembling known folds at low sequence identity, with higher folding confidence and closer structural matches to the PDB than a decoder-only baseline.

    From 14,626 generated sequences kept after filtering (nucleus sampling over top-P 0.5-1.0 and temperature 0.2-1.0, perplexity filter, MMseqs2 clustering), ESMFold mean pLDDT was 84.0 against 74.3 for 8,466 PROGEN2-xlarge sequences, and mean Foldseek TM-score to PDB 0.695 against 0.522, at identity 0.224 against 0.165. For the 11,048 sequences with pLDDT above 80, mean max TM-score to AlphaFold/UniProt50 was 0.864. · xTrimoPGLM Generate Sequences with Diverse…

  4. ShownOn held-out sequences deposited after the training cutoff, perplexity was lower than for larger-token or decoder baselines, and was reached with fewer training tokens.

    On two sets of about 10,000 UniProt sequences filtered at 0.5 and 0.9 identity to the training data, the model scored 10.81 and 6.70 against ESM2-15B's 10.98 and 7.25, and 13.35 and 8.78 against PROGEN2-xlarge's 14.30 and 9.75. The masked-objective match to ESM2 (trained on 1 trillion tokens) came within 480 billion tokens, the generative match to PROGEN2-xlarge (350 billion) within 120 billion. · xTrimoPGLM Achieves Low Perplexity on…

  5. SuggestedDownstream performance generally tracked pre-training compute, but not for every task: a minority got worse as compute grew.

    Exponential growth in pre-training compute (PF-days) mapped to linear gains in task performance for both probing and LoRA fine-tuning. Against the masked-objective validation loss, 44% of tasks improved gradually, 28% showed a sudden late jump the authors call grokking, 16% were non-linear, and 12% degraded, an inverse scaling effect. The supporting figures are not in the extracted text. · Comparison Between ESM and xTrimoPGLM Family…; Comparison Among Different Training States of…

Methods, data and reproducibility

A General Language Model backbone (72 layers, 80 heads, 10,240 embedding dimensions) is trained on residue tokens with two objectives marked by distinct tokens: masked prediction over a bidirectional prefix and autoregressive span infilling. Training ran 400 billion tokens of the masked objective, then 600 billion at 20:80. Downstream use is MLP probing or LoRA fine-tuning, plus a 90M-parameter folding trunk.

  • Pre-training corpus from Uniref90 (version before December 2022) and ColabFoldDB, deduplicated, length- and composition-filtered, sequences published before January 1, 2023 · multiple species · n = ~939M sequences, 200B residue tokens
  • Validation set of UniProt sequences released between January 1 and March 30, 2023, filtered with MMseqs2 against Uniref50 and the training data · multiple species · n = 18M sequence increment before filtering
  • Two out-of-distribution perplexity sets from UniProt after the training cutoff, at 0.5 and 0.9 identity to training data · multiple species · n = ~10,000 sequences each
  • Structure prediction test sets CAMEO (released April-June 2022) and CASP15 (released May 2022) · multiple species · n = 194 and 56 proteins
  • 18 downstream benchmark datasets over protein structure, developability, interactions and function, e.g. trRosetta contact maps with the Ankh split · multiple species · n = contact map: 12,041 train / 1,505 valid / 1,505 test
  • Generated sequences evaluated for structure, compared with sequences from PROGEN2-xlarge · none · n = 14,626 and 8,466 sequences
  • Supervised fine-tuning and reinforcement self-training sets drawn from five of the benchmark tasks plus four SCOP fold types · multiple species
Methods & tools
xTrimoPGLM, xT-Fold, ESM2 (150M, 3B, 15B), ESMFold, OmegaFold, AlphaFold2, PROGEN2-xlarge, ProtGPT2, LoRA, MMseqs2 and HHblits, Foldseek, DeepSpeed v0.6.1
Reagents & environment
DeepSpeed v0.6.1 for training; analysis with Python v3.8, NumPy v1.16.4, SciPy v1.2.1, seaborn v0.11.1, Matplotlib v3.3.4, pandas v1.1.5, TM-align v20190822, PyMOL v2.3.0. FP16 with Apex O2 mixed precision, AdamW, 2,048-token samples with mixed context windows.
Assay conditions & compute
Pre-trained on 96 DGX-A100 servers (8 x 40G each) from 18 January to 30 June 2023, consuming 1 trillion tokens, with 4-way tensor, 8-way pipeline and 24-way data parallelism. Fine-tuning without quantisation needs at least four 80G A100s; quantised xT-Fold inference runs on one A100.
Reproducibility gaps
Weights, downstream datasets and the task-dataset collection are said to be available, but every address was lost in text extraction, so no link can be listed; the only usable address left points at a baseline (ESMFold), not this work. Per-task score tables and all figures are also missing.
Limitations (6)
  • Single-sequence modelling still trails MSA-based and retrieval-augmented methods on out-of-distribution targets such as CASP proteins, and the authors report diminishing returns from further scaling on that gap. · stated by the authors
  • Generation hallucinates: low sampling temperature yields repetitive amino-acid stretches that can fold confidently in silico but do not occur naturally, about 20% of generated sequences fall below pLDDT 70, and after fine-tuning only 17.8% to 88% of sequences matched a PDB structure. · stated by the authors
  • Compute cost is a barrier to use: fine-tuning without quantisation needs at least four A100 80G GPUs. · stated by the authors
  • 12% of downstream tasks got worse with more pre-training compute, so scaling is not uniformly beneficial. · stated by the authors
  • Fine-tuned generation is judged by the study's own task predictors as in silico evaluators, with no experimental validation, and the authors call these biased evaluators. · noted in this summary
  • Several comparisons mix reproduced and cited numbers, and where no benchmark existed the authors' own ESM2-15B with LoRA stands in as the reference, so the 15-of-18 claim is partly against a self-generated baseline. · noted in this summary

Who should read:Read this if you weigh protein language models for both prediction and design, or want evidence on how pre-training scale transfers to downstream protein tasks and single-sequence folding.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISPourmirzaei '24Pourmirzaei '25Garibbo '26Bhatnagar '25Ferruz '22Su '23
78% semantic matchBioinformatics
Mahdi Pourmirzaei, Farzaneh Esmaili, Mohammadreza Pourmirzaei+2 more·bioRxiv·May 31, 2024· 9 citations