henhaobio
Sign in

bioRxivGenomicsPreprint

Evo 2's Perception of Single Nucleotide Substitutions in the Genes of Two Plant Model Organisms

Otho MantegazzaiD, Lorenzo BertoliniiD, Gabriele LeoniiD, Moreno ColaiacovoiD, Mauro PetrilloiD, Laura BonfiniiD, Cristian SAVINI, Mario CeresaiD, Xavier ZaouiiD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Although DNA Large Language Models (DNA-LLMs) offer a path to decoding genetic complexity, our ability to evaluate these models is constrained by our incomplete understanding of the very same genetic syntax and functional logic that these models are trained to learn. In this study we use single nucleotide substitutions that have or have not been observed in living organisms, to evaluate how the DNA-LLM Evo 2 interprets gene sequences from two plant model organisms, Arabidopsis thaliana and Oryza sativa japonica. Using perplexity as a measure of the models confidence, we observe that alleles containing simulated substitutions are perceived, on average, as less likely than those observed in vivo. Although the size of the effect is modest, the effect is statistically significant and robust, suggesting that Evo 2 is aligned with our current understanding of evolutionary selective constraints. This approach is designed to be model-agnostic and species-agnostic and could serve as a generic framework for evaluating the performance of DNA-LLMs.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (21 Jul 2026) · CC BY 4.0

Evo 2 7B assigns higher perplexity to plant gene alleles carrying randomly simulated single-nucleotide substitutions than to those observed in vivo, in Arabidopsis and rice; significant but small effect.

Evaluating DNA language models is hard because the genetic syntax they learn is itself incompletely understood, so the cognition-style benchmarks used for text LLMs cannot easily be applied. Compared with the substitution tests used by Brixi et al. (2026) to assess Evo 2 and those in DART-Eval, this framework covers fewer species but relies on random sampling to keep assumptions about mutations minimal and to widen the range of mutations tested.

Key results

  1. ShownWithin genes, Evo 2 7B gives alleles with simulated substitutions higher perplexity than alleles with variants observed in vivo, in both species, for non-coding, silent and missense classes. The separation is small, and the distributions overlap heavily.

    Per-gene one-sided Wilcoxon-Mann-Whitney p-values are strongly right-skewed rather than uniform. Median probability of superiority in Arabidopsis thaliana: 0.56 non-coding, 0.57 silent, 0.61 missense. In Oryza sativa japonica: 0.58, 0.60, 0.69. Nonsense: widely dispersed, no detectable effect. · 3.7 Simulated mutations induce higher perplexity…; Figure legends

  2. ShownSubstituted alleles, whether observed or simulated, usually score higher perplexity than the unmodified reference allele of the same gene. The shift is larger for simulated substitutions.

    Median per-gene share of substituted alleles above reference perplexity (50% expected if random): Arabidopsis thaliana 62.7% (bootstrap CI 61.2%-64.4%) observed, 74.2% (72.6%-75.4%) simulated; Oryza sativa japonica 61.7% (60.0%-62.9%) observed, 76.8% (75.4%-78.1%) simulated. · 3.6 Reference alleles have lower perplexity than…; Figure legends

  3. ShownOn unmodified genes, Evo 2 7B predicts the next nucleotide in coding sequence with near-certain probability but near chance in UTRs and introns. There are minor reciprocal peaks, and an unexplained bimodal peak in non-coding regions.

    Across all unmodified genes in both species, correct-nucleotide probabilities peak near 1 for protein-coding sequence and near 0.25 (chance among four bases) for non-coding sequence. A Kolmogorov-Smirnov test indicates the two distributions differ. Examples shown: AT1G03190 and Os09g0567366. · 3.3 Evo 2 accurately predicts the next…; Figure legends

  4. ShownPerplexity varies far more between genes than between single-nucleotide variants of the same gene, so all later comparisons were kept within genes. Within-gene spread shrinks as gene length increases.

    Intraclass correlation coefficient (gene as random intercept) is approximately 1.0 in both species. Within-gene perplexity IQR vs gene length: Spearman ρ −0.90 in Arabidopsis thaliana and −0.94 in Oryza sativa japonica. Median IQR 0.0045 (Arabidopsis thaliana) vs 0.0032 (Oryza sativa japonica). · 3.5 Perplexity varies more among different genes…; Figure legends

  5. SuggestedA missense substitution lowers Evo 2's probability of the correct next nucleotide both at the site and downstream, more for simulated than for observed substitutions. The effect decays roughly exponentially but can persist for hundreds of nucleotides.

    Probability differences (substituted minus reference) over the site and the 25 downstream nucleotides are shifted negative on both strands, in a pseudorandom 10% subsample of genes. Per Table 2, the effect stays detectable for several hundred nucleotides and occasionally beyond 1000. · 3.8 Single nucleotide substitutions affect Evo…; Figure legends; Tables

Methods, data and reproducibility

From Ensembl Plants 60, 500 genes per species were sampled at random (432 Arabidopsis and 499 rice protein-coding genes kept). Alleles were built for every recorded single-nucleotide variant plus all three alternatives at a random 5% of positions. Whole-gene sequences were scored with Evo 2 7B (perplexity averaged over forward and reverse-complement strands) and compared within genes using nonparametric tests.

  • Protein-coding gene sequences from the TAIR10 reference genome (Ensembl Plants 60), from earliest TSS to latest TTS · Arabidopsis thaliana · n = 432 genes (of 27628)
  • Protein-coding gene sequences from the IRGSP-1.0 reference genome (Ensembl Plants 60), from earliest TSS to latest TTS · Oryza sativa japonica · n = 499 genes (of 35804)
  • Observed single-nucleotide variants from the Ensembl Plants 60 variation database (VCF files) · Arabidopsis thaliana; Oryza sativa japonica
  • Single-nucleotide substituted gene sequences (observed variants plus simulated substitutions at a random 5% of positions) · Arabidopsis thaliana; Oryza sativa japonica · n = 351894 (A. thaliana) and 468984 (O. sativa japonica) sequences · 10.2905/JRC.JH9JHTG
Methods & tools
Evo 2 7B, Evo2 Python library, PyTorch, Ensembl Plants 60, R 4.6.0, RENV 1.0.11, Bioconductor 3.23, Quarto 1.9.16, reticulate 1.46.0, lme4 2.0-1, performance 0.16.0, rsample 1.3.2
Software & environment
R 4.6.0 with package versions recorded by RENV 1.0.11; Bioconductor 3.23; Tidyverse 2.0.0. Manuscript can be regenerated with Quarto 1.9.16. Evo 2 run through the official Evo2 Python library with default settings; preprocessing via the R/preprocess.R wrapper.
Assay conditions & compute
Evo 2 7B inference ran on NVIDIA H100 80GB GPUs. All other steps are stated to run on standard consumer-grade hardware.
Reproducibility gaps
No versions are given for Python, PyTorch or the Evo2 library used for inference. GPU count, runtime and memory use are not reported.
Limitations (6)
  • Effect sizes are modest. Perplexities of simulated and observed alleles overlap so much that perplexity alone cannot tell which kind a randomly drawn allele is. · stated by the authors
  • No effect was detected for nonsense substitutions. The authors attribute this to the small number of nonsense variants that a single base change can produce. · stated by the authors
  • Recorded variants are only a sample of those that occur in nature. Observed alleles scoring above reference could reflect rare variants or overfitting to reference genomes, and more data on natural variation and allele fitness are needed to tell these apart. · stated by the authors
  • Only two plant species are studied, fewer than in earlier Evo 2 evaluations. Because the model sees sequence alone, what its likelihoods mean biologically (mutation probability versus fitness) remains unresolved. · stated by the authors
  • The framework is described as model-agnostic, but only Evo 2 7B is evaluated. There is no second DNA-LLM and no non-model baseline for comparison. · noted in this summary
  • The decay analysis covers only missense substitutions in a 10% gene subsample, partly by visual inspection. The species comparison is unclear: the main text says observed-mutation effects appear more persistent 'particularly in Oryza sativa japonica', but the Table 2 caption says decay is slightly faster there. · noted in this summary

Read this if you want a simple, assumption-light check of whether a DNA language model's sequence likelihoods track recorded natural variation in plant genes, or are considering Evo 2 perplexity for scoring plant variants.

Citations
1

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISZhai '24Zhai '26Minseo '26Consens '26Weber '18Bertolini '26
51% semantic matchBioinformatics
Jingjing Zhai, Aaron Gokaslan, Yair Schiff+9 more·bioRxiv·Jun 4, 2024· 18 citations