Ankh : Optimized Protein Language Model Unlocks General-Purpose Modelling
Ahmed ElnaggariD, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte RochereauiD, Burkhard RostiD
Abstract
As opposed to scaling-up protein language models (PLMs), we seek improving performance via protein-specific optimization. Although the proportionality between the language model size and the richness of its learned representations is validated, we prioritize accessibility and pursue a path of data-efficient, cost-reduced, and knowledge-guided optimization. Through over twenty experiments ranging from masking, architecture, and pre-training data, we derive insights from protein-specific experimentation into building a model that interprets the language of life, optimally. We present Ankh, the first general-purpose PLM trained on Googles TPU-v4 surpassing the state-of-the-art performance with fewer parameters (<10% for pre-training, <7% for inference, and <30% for the embedding dimension). We provide a representative range of structure and function benchmarks where Ankh excels. We further provide a protein variant generation analysis on High-N and One-N input data scales where Ankh succeeds in learning protein evolutionary conservation-mutation trends and introducing functional diversity while retaining key structural-functional characteristics. We dedicate our work to promoting accessibility to research innovation via attainable resources.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 186
- Organism
- Assay
- Research Focus
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 1 (18 Jan 2023) · CC BY-NC-ND 4.0
Protein-specific tuning of masking, depth and positional embeddings, rather than more parameters, yields an encoder-decoder model that tops structure and function benchmarks at a fraction of the compute.
Protein language models (PLMs) have grown from about 10^6 to 10^9 parameters on the assumption that bigger is better, which pushes the cost of entry out of most labs' reach. The authors argue scale is conditional and run over twenty single-variable experiments on masking, architecture and pre-training data to build a smaller model instead.
Key results
ShownAnkh surpassed the previous best PLM results across the structure, function and localization benchmarks, with Ankh base matching or coming close, while both train and run far smaller.
Average improvement over the state of the art of 4.8% for Ankh and 3.4% for Ankh base, with under 10% and 3% of the training parameters and 30% and 15% of the embedding dimension; the abstract also cites under 7% of inference parameters. Per-task scores sit in tables absent from this text, and the baseline is ProtT5-XL-U50 (3B training, 1.5B inference parameters). · Abstract; 2.1 Utmost Computational Efficiency and Utmost…; Results Summary - More Efficient Models Can Also…
ShownThe efficiency claim is backed by feature-extraction timings and hardware requirements rather than parameter counts alone.
For sequences up to 1024 residues, ESM-2 (15B) needed at least 2.2x and 2.0x, and at most 11.7x and 7.1x, the feature-extraction time of Ankh base and Ankh. ESM-2 (15B) required four A100 80GB GPUs for feature extraction versus a single A100 40 GB for each Ankh model. · 2.1 Utmost Computational Efficiency and Utmost…
SuggestedScale did not track performance among the comparison models: the largest ESM-2 lost to smaller ESM models on some tasks and varied between runs.
ESM-2 (15B) did not outperform smaller models of the ESM family on all tasks and showed inconsistent results across runs. The authors also found embedding-based prediction beat attention-based prediction for contact prediction, contrary to what is reported for the ESM family. No run-to-run variance figures appear in this text. · 2.1 Utmost Computational Efficiency and Utmost…; 7.3.3 Attention-Based Predictions
ShownThe final configuration was assembled greedily from single-variable ablations, each trained for 2 epochs, which settled masking, layer split, positional embeddings and training set.
Chosen: 1-gram span partial de-masking (Exp. 4), 20% masking probability (Exp. 8), 48 encoder and 24 decoder layers at constant 72 total (Exp. 11), Gated-GELU, relative positional embedding offset 128 with dimension 64 (Exp. 20), weight tying rejected (Exp. 22), and UniRef50 retained over UniRef90 (Exp. 23). Ankh base uses embedding dimension 768 and Ankh 1536. · 2.2.3 Knowledge-Guided Optimization in Action; a. Number of Encoder-Decoder layers; d. Relative Positional Embedding; 7.1 Pre-Training Data & Model Experimentation
SuggestedIn generation tests the model reproduced family-level conservation patterns and, with higher sampling temperature, reached rare functional subfamilies; single-sequence variants kept predicted folds despite low sequence identity.
Fine-tuned on malate dehydrogenase (MDH), sets of 500 sequences at temperatures 1.0, 1.5 and 2.0 matched the natural alignment's Shannon entropy with mean squared error 0.1, 0.09 and 0.08, using under 3% of the 16,706 natural sequences; over 95% of variants were unique, identities fell as low as 70% and 55%. A rare CATH functional family (6 natural occurrences) appeared only at temperature 2 (155 and 95 occurrences). · 2.2.1 Auto-Regressive Fine-Tuning for High-N…; 2.2.2 Masked Language Modeling (MLM) for One-N…; 4.4 Variant Generation Model
Methods, data and reproducibility
An encoder-decoder (T5-style) transformer is pre-trained on UniRef50 on Google TPU v4 pods using Flax/JAX, with design choices fixed by one-variable-at-a-time experiments. Frozen embeddings feed a shared ConvBERT-plus-linear head for every downstream task; variants are generated by auto-regressive fine-tuning of the decoder (family scale) or masked-language-model inference (single sequence).
- UniRef50 used for pre-training, with UniRef90 tried as an alternative; UniRef100 and BFD cited as previously tested
- Function benchmarks: green fluorescent protein mutant fluorescence (TAPE splits), DeepSol solubility, GB1 binding fitness from FLIP · n = 149,361 GB1 mutants down-sampled to 8,733
- Structure benchmarks: ProteinNet contact prediction with CASP12 test plus CASP14 free-modeling, Hou fold prediction, NetSurfP-2.0 secondary structure tested on CB513, TS115, CASP12 and CASP14 · n = 1194 folds
- Embedding-based annotation transfer benchmark curated from CATH v4.3, and DeepLoc sub-cellular localization · n = 69k lookup and 219 test sequences; 10 localization classes
- Malate dehydrogenase (MDH) variant set used by ProteinGAN, for family-based generation · n = 16,706 unique sequences
- Single-chain SARS-CoV-2 nanobodies added to CoV-AbDab after June 2022, for one-shot generation · n = seven nanobodies; 179 synthetic variants
- Methods & tools
- Ankh / Ankh base, ProtT5-XL-U50, ESM-1b, ESM-2, ConvBERT, Flax / JAX, TPU v4, ColabFold, BioPython (pairwise2, SVDSuperimposer), HuggingFace, UniRef, CATH
- Reagents & environment
- Models built in Flax/JAX and released on HuggingFace for JAX, TensorFlow and PyTorch users, under CC BY-NC-SA 4.0; the repository is said to include Python and notebook tutorials runnable on free online resources.
- Assay conditions & compute
- Pre-training on Google TPU v4 pods with either 64 or 128 cores (8 cores per host, 16 GiB per core, mega-core merging giving 32 GiB); downstream feature extraction on a single A100 40 GB GPU.
- Reproducibility gaps
- The availability section's repository and licensing links are missing from this extracted text, so no code or model URL can be given here; benchmark result tables, pre-training data statistics and model configuration tables are likewise absent.
Limitations (5)
- Because Gated-GELU carries many trainable parameters, width had to be traded against depth to hold parameter count constant, so activation and dimensionality were not cleanly separable. · stated by the authors
- Design choices were propagated greedily task by task: no configuration won everywhere, and the authors admit to no single formula defining 'best'. · stated by the authors
- Every ablation variant was trained for only 2 epochs (one for the UniRef90 run), so the rankings may not hold after full training. · stated by the authors
- Benchmark numbers are summarised only as averaged percentage gains in this text; the per-task tables are missing, so the size of each improvement cannot be judged here. · noted in this summary
- Generated variants were assessed in silico, against predicted structures and alignment statistics, with no experimental test of activity. · noted in this summary
Who should read:Read this if you pick protein language models for downstream tasks on a limited GPU budget, or want evidence on which pre-training choices (masking, layer split, positional embeddings) actually move protein benchmarks.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- TooT-PLM-P2S: Incorporating secondary structure information into protein language models
Hamed Ghazikhani, Gregory Butler·bioRxiv·Aug 13, 2024
- Protein Language Models: Is Scaling Necessary?
Quentin Fournier, Robert McCoy Vernon, … CHRISTOPHER JAMES LANGMEAD·bioRxiv·Sep 23, 2024
- Fine-tuning protein language models boosts predictions across diverse tasks
Robert Schmirler, Michael Heinzinger, Burkhard Rost·Nature CommunicationsPeer-reviewed·Aug 28, 2024
- E1: Retrieval-Augmented Protein Encoder Models
Sarthak Jain, Joel Beazer, … Ali Madani·bioRxiv·Nov 12, 2025
- xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
Bo Chen, Xingyi Cheng, … Le Song·bioRxiv·Jul 5, 2023
- AINN-P1: A Compact Sequence-Only Protein Language Model Achieves Competitive Fitness Prediction on ProteinGym
Roger W. Wang, Kevin Jin, Lurong Pan·bioRxiv·Mar 30, 2026