henhao
Sign in
BioinformaticsbioRxiv
Preprint

A Taxonomy-Informed Sparse DNA Foundation Model for Microbial Genomics

Athan Z. LiiD, Shiyuan WangiD, Shupeng ChengiD, Yuxuan DuiD, Ruishan LiuiD

Abstract

Microorganisms are crucial to Earth's ecosystems, with genomes encoding functions important to agriculture, biotechnology and human health. Although genomic language models have advanced DNA representation learning, the extensive diversity of microorganisms and highly imbalanced taxonomic composition of existing pretraining corpora pose challenges for effective microbial genomic sequence modeling. Here we present MicroGlot, a taxonomy-informed microbial DNA foundation model pretrained on 3.70 million sequences comprising 378.3 billion nucleotides from 99,700 species. MicroGlot represents hierarchical relationships among taxa using hyperbolic embeddings and integrates this knowledge into a sparse mixture-of-experts architecture. Probing frozen embeddings across layers showed that MicroGlot captures both microbial phenotypic traits and taxonomic identity. Compared with a taxonomy-ablated variant pretrained under the same scheme, incorporating taxonomic knowledge consistently improved representation quality across model depth. Combining sparse computation with efficient architectural and training techniques, MicroGlot achieved competitive probing and fine-tuning performance with low computational requirements. Species-level expert-routing fingerprints aligned more closely with taxonomic groups than tetranucleotide-frequency profiles, indicating that expert usage recapitulates taxonomic structure. These results show that combining sparse architecture with taxonomic knowledge enables efficient and biologically informative genomic language modeling across diverse microbial taxa. MicroGlot is publicly available at https://huggingface.co/athanzli/MicroGlot.

PDFDOICodeData
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
0
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (28 Sept 2026) · CC BY 4.0

A sparse mixture-of-experts DNA model conditioned on hyperbolic taxonomy embeddings learns better microbial representations than its taxonomy-free twin and comes close to Evo2-7B in probing and fine-tuning at lower compute.

Microbial genomes are phylogenetically diverse and unevenly represented in pretraining corpora, straining the shared parameters of genomic language models, and earlier species conditioning (learned embeddings in SpeciesLM and SPACE, lineage tokens in Evo and Evo2) leaves embedding geometry unconstrained by the taxonomic tree. MicroGlot encodes that tree as hyperbolic Poincaré embeddings and uses them as an input token and to steer sparse mixture-of-experts routing.

Key results

  1. ShownAdding the taxonomy inputs (a species token plus species-modulated expert routing) improved the model over an otherwise identical taxonomy-free twin (MicroGlot-plain), in pretraining perplexity and in frozen-probe performance across nearly every layer and task.

    Validation perplexity 67.45 vs 68.99 for MicroGlot-plain, lower at all 22 checkpoints; better best-layer and layer-averaged scores on all 17 tasks (7 traits, 10 yeast histone marks) and in 370 of 391 task-layer comparisons; on histone marks with one fixed S. cerevisiae embedding, mean layer-averaged MCC 0.552 vs 0.531. · 2.4 Taxonomic knowledge improves MicroGlot…; 4.5 Pretraining

  2. ShownIn frozen-embedding probes, the MicroGlot variants led layer-averaged scores on microbial trait and taxonomic tasks. Evo2-7B-base edged ahead when each model was scored at its validation-selected best layer. MicroGlot used the fewest pretraining tokens of all models compared.

    Layer-averaged: MicroGlot best on all 7 trait tasks, MicroGlot-plain best on 4 of 6 taxonomic tasks; 13-task mean 0.816 vs 0.798 for ProkBERT-mini-long (top baseline). Best layer: Evo2-7B-base 0.841 vs 0.835 for the MicroGlot pair. Pretraining: 70.6 billion tokens, ~2.03×10^20 FLOPs, 479 million active parameters. · 2.3 MicroGlot encodes microbial taxonomy and…; 4.7 Probing

  3. ShownUnder identical LoRA settings on 13 GUE tasks, MicroGlot needed the fewest epochs and much less compute than Evo2-7B-base. MicroGlot-plain (taxonomy-free) had the top mean macro F1 on the three taxonomic tasks and the top viral species score, but MicroGlot trailed Evo2-7B-base on the 13-task mean.

    Taxonomic tasks mean macro F1 0.689 vs 0.686 (Evo2-7B-base, DNABERT-2) and 0.668 (ProkBERT-mini-long); viral species 0.448 vs 0.443. 13-task mean 0.627 vs 0.657 for Evo2-7B-base; 2.31 epochs on average; ~134-fold less fine-tuning compute than Evo2-7B-base (~3.73 × 10^18 FLOPs per task). · 2.5 MicroGlot enables efficient genomic…; 4.8 Fine-tuning

  4. ShownBackbone inference (synthetic inputs, batch size one, one GPU) used less memory and time than the encoder baselines and Evo2-7B-1M at comparable lengths. Memory stayed nearly flat up to the trained context, and forward passes ran far beyond it.

    At 8,192 tokens (~43.4 kb): 6.27 GB and 80.6 ms vs 9.88 to 27.63 GB and 244 to 1,143 ms for encoders at 16.4 to 49.1 kb, and 32.57 GB and 3.27 s for Evo2-7B-1M at 32.8 kb; throughput 538 vs 123 to 364 kb/s; ran 2,097,152 tokens (11.1 Mb) in 76.7 GB vs 131 kb maximum for any other model. · 2.5 MicroGlot enables efficient genomic…

  5. ShownExpert routing follows taxonomy. Per-species routing fingerprints align with broad taxonomic groups far more closely than tetranucleotide-frequency (TNF) profiles from the same windows, while individual experts remain shared across groups.

    1,000 species (250 each from Bacteria, Archaea, Eukaryota and viruses), 3,948 8 kb windows from 86 phyla; k-means with 9 clusters: adjusted Rand index 0.620 ± 0.056 vs 0.115 ± 0.013 for TNF over ten seeds; mean cosine similarity 0.295 within vs −0.063 between reference groups. · 2.6 MicroGlot learns multilingual…; 4.9 Sequence composition and expert routing…

Methods, data and reproducibility

A 23-layer decoder-only transformer with residual mixture-of-experts layers (~2.98B parameters, 479M active per token) was pretrained by next-token prediction on BPE-tokenized DNA; 32-dimensional Poincaré embeddings of an eight-rank taxonomy enter as a species token and modulate expert routing. Evaluation used frozen-layer probes, LoRA fine-tuning, inference benchmarks and a taxonomy-ablated twin.

  • Pretraining corpus: RefSeq release 232, ICTV Virus Metadata Resource MSL40 v1 and PLSDB 2024_05_31_v2 plasmids, taxonomy harmonized across eight ranks (NCBI Taxonomy snapshot of 1 September 2025) · bacteria, archaea, fungi, protists, viruses; plasmids carry host taxonomy · n = 3,695,065 sequences; 378.3 billion nucleotides; 99,700 species · RefSeq release 232; VMR MSL40 v1; PLSDB 10.6084/m9.figshare.27252609.v2
  • geNomad reference collection, class-balanced for virus, plasmid and chromosome discrimination (3 probing tasks) · viruses, plasmids, prokaryotic chromosomes · n = 121,769 sequences after deduplication · 10.5281/zenodo.8049246
  • Viral family classification set: gut virome sequence clusters of known families, obtained from the Virgo benchmark archive · viruses · n = 473 sequences, 8 families · 10.6084/m9.figshare.28730093.v1
  • Global Ocean Eukaryotic Viral database assemblies, distributed with Virgo (giant virus order classification) · giant viruses · n = 1,370 assemblies, 4 orders · 10.6084/m9.figshare.20284713.v11
  • 12 DNABERT-S datasets including CAMI II simulations (marine, plant-associated and synthetic community species classification) · n = 110,492 sequences, 1,208 labels (809 OTUs)
  • BactoTraits species-level annotations matched to RefSeq assemblies: Gram stain, motility, sporulation, oxygen tolerance, temperature and pH preference · microbial species from 35 phyla · n = 3,094 genomes (3,020 species); 890 to 2,899 per trait · 10.24396/ORDAR-182
  • Phydon minimum doubling times matched to GTDB release 220 assemblies (growth rate regression) · bacteria and archaea · n = 1,027 genomes; 496 species from 41 phyla
  • GUE benchmark: 10 yeast histone-mark datasets (500 bp), SARS-CoV-2 variants (9 classes), viral species (25) and fungal species (20) · S. cerevisiae; SARS-CoV-2; viruses; fungi · n = 13 datasets
Methods & tools
PyTorch (FlashAttention-2 backend), Poincaré embeddings (facebookresearch reference implementation), Hugging Face tokenizers v0.21.4, Transformers v4.51.3, DeepSpeed v0.18.4 (ZeRO stage 1), PEFT v0.13.2 (LoRA), CD-HIT-EST v4.8.1, PSI-CD-HIT with BLAST+ v2.17.0, UMAP, GTDB release 220, Baselines: Evo2, DNABERT-2, DNABERT-S, ProkBERT-mini-long, Nucleotide Transformer, GUE benchmark
Code
https://github.com/athanzli/MicroGlothttps://huggingface.co/athanzli/MicroGlot
Data & accessions
https://ftp.ncbi.nlm.nih.gov/refseq/release/https://ictv.global/sites/default/files/VMR/VMR_MSL40.v1.20250307.xlsxhttps://doi.org/10.6084/m9.figshare.27252609.v2https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2025-09-01.ziphttps://doi.org/10.24396/ORDAR-182https://drive.google.com/file/d/1uOrwlf07qGQuruXqGXWMpPn8avBoW7T-/view
Software & environment
Tokenizer released for tokenizers v0.21.4 and transformers v4.51.3; training with Transformers v4.51.3, DeepSpeed v0.18.4 ZeRO stage 1, bfloat16 with float32 routing; PEFT v0.13.2 for LoRA; CD-HIT-EST v4.8.1 and BLAST+ v2.17.0 for deduplication.
Assay conditions & compute
MicroGlot, MicroGlot-plain and the species encoder were each trained one epoch on 8 NVIDIA RTX PRO 6000 Blackwell GPUs (~5 days; 22,087 steps; 70.56 billion tokens); MicroGlot pretraining ~2.03×10^20 FLOPs. Inference: 6.27 GB at 8,192 tokens on one RTX PRO 6000 Blackwell Max-Q.
Reproducibility gaps
Per-task benchmark preparation is in Supplementary Note 1, and full model, pretraining and LoRA configurations are in Supplementary Tables 1 to 3; none of these is in the provided text. Viral family labels come from a metadata table deposited with the source code.
Limitations (4)
  • Although the reference set holds 99,700 species, the authors note it covers only a fraction of estimated global microbial diversity. They propose expanding the corpus and the species embedding set as RefSeq and ICTV taxonomy grow. · stated by the authors
  • The trained context is 8,192 tokens (~43.4 kb). The authors point to staged pretraining on progressively longer sequences, as used for Evo2, as the route to longer-range genomic dependencies such as distal regulatory interactions. · stated by the authors
  • In trait probing, MicroGlot receives each genome's taxonomy-derived species embedding, the traits are species-level labels, and the baselines see only sequence. Part of its trait advantage may therefore come from that input; the yeast histone control holds species fixed, but only within one species. · noted in this summary
  • Routing fingerprints were computed with each window's annotated species embedding, which directly modulates the routing logits. Their alignment with taxonomic groups (ARI 0.620 vs 0.115 for TNF) is therefore partly built in, and no MicroGlot-plain routing comparison is reported. · noted in this summary

Who should read:Read this if you build or apply DNA language models to microbial or viral genomes and want a compute-efficient sparse model, or want a worked example of injecting hierarchical taxonomy into pretraining and expert routing.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISFang '26Narayan '26Hoarfrost '22Bishop '26Pope '23Leske '25
65% semantic matchBioinformatics
Chao Fang, Fangming Yang, Hao Hou+31 more·bioRxiv·May 24, 2026· 3 citations