Alignment-Aware Genomic Coordinate Encoding for Long-Range DNA Modeling

Abstract

Existing genomic foundation models (GFMs) are adept at capturing local, sequential dependencies in DNA. However, their architectures often struggle to efficiently model gene regulation features, such as sparse motifs and distal regulatory elements that operate beyond linear proximity. Moreover, standard positional encodings fail to fully exploit the reference genome coordinate system. Therefore, we present USM-AA, an alignment-aware genomic foundation model that incorporates reference genome coordinates through a non-contiguous positional encoding, Alignment-Aware Rotary Position Embedding, and an alignment embedding module encoding chromosome, strand, and Compact Idiosyncratic Gapped Alignment Report (CIGAR) information. To improve long-range sequence modeling, USM-AA leverages a reverse-complement masked learning strategy. Pretrained on alignment-aware genomic sequences spanning cross-species evolution and human population variation, USM-AA consistently outperforms existing models across multiple downstream tasks, highlighting the importance of alignment-aware representations for genomic sequence modeling.