Benchmarking DNA Foundation Models: Biological Blind Spots in Evo2 Variant-Effect Prediction

Abstract

DNA foundation models such as Evo2 and DNABERT-2 show promise for variant-effect prediction (VEP), yet rigorous benchmarks assessing their ability to capture established biological constraints remain limited. Here, we introduce controlled benchmarks based on well-characterized nuclear and mitochondrial sequence features. Although Evo2 achieves strong aggregate performance on mitochondrial variants (AUROC = 0.927), our analyses reveal systematic blind spots in short-range signals, including codon usage bias, as well as spurious sensitivity to biologically neutral sequence context. Notably, changes in flanking sequences can cause pathogenic tRNA variants to be misclassified as benign. These findings suggest that Evo2's predictive performance may rely more heavily on learned sequence regularities than on the biological constraints governing variant function, raising concerns about the clinical utility of zero-shot genomic foundation models.Our framework provides a principled basis for standardized model evaluation and for guiding the development of architectures better aligned with biological constraints.