henhaobio
Sign in

bioRxivGenomicsPreprint

ARSENAL: Learning Transferable Regulatory DNA Representations with Targeted Short-Context Language Models

Aman PateliD, Anshul Bharat KundajeiD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

DNA language models (DNALMs) aim to learn representations of genomic sequence for variant interpretation, regulatory prediction, and sequence design. Most DNALMs are trained on whole genomes and long contexts, but regulatory DNA poses a distinct challenge: functional elements are sparse, context dependent, and encoded by short transcription factor motif syntax embedded in extensive background sequence. We introduce ARSENAL, a short-context masked DNA language model pretrained on ENCODE candidate cis-regulatory elements. ARSENAL recovers diverse transcription factor motifs de novo and improves zero-shot regulatory variant effect prediction relative to other DNALM foundation models. ARSENAL embeddings also improve supervised regulatory sequence models at predicting chromatin accessibility and regulatory variant scoring. Finally, ARSENAL serves as an efficient generative prior, enabling multi-objective regulatory sequence design with supervised oracles. ARSENAL shows that targeted self-supervised pretraining on regulatory regions can learn biologically meaningful and transferable regulatory representations without genome-scale training, long contexts or task-specific labels. Code is available at https://github.com/kundajelab/regulatory_lm. Models and data are shared at https://sageb.io/ydjhqM

In-depth summary

Claude Sonnet 4.6 · xhigh effort · 7 Nov 2026 · from the full text of version 3 (12 Jul 2026) · CC BY 4.0

A 350 bp masked DNA language model trained on ENCODE cCREs with a frequency-domain regularizer recovers TF motifs zero-shot, scores regulatory variants, improves ChromBPNet inputs, and guides sequence design.

The paper says whole-genome DNA language models often do poorly on regulatory tasks such as variant effect prediction, because regulatory motif signal is sparse against background sequence. ARSENAL instead uses a short context, pretraining on regulatory elements only, and a Fourier-based loss that favours motif-scale structure, without alignments or labels.

Key results

  1. ShownARSENAL recovers diverse transcription factor motifs from zero-shot likelihoods. On the DART-EVAL motif identification benchmark it beats other DNA language models. The ablation without the Fourier regularizer still beats the other models but trails the full model.

    DART-EVAL motif benchmark (Fig. 3A), no numeric scores given in text. TF-MoDISco found NRF, NFY and ETS motifs in K562 PRO-cap promoters, and HNF, TEAD and MEF2 in HepG2 DNase peaks. · 2.2 ARSENAL improves zero-shot motif discovery; 4.4 Zero-Shot Visualization and Motif Discovery

  2. ShownZero-shot variant scoring with ARSENAL correlates better with measured effect sizes than other DNA language models on the DART-EVAL dsQTL and caQTL sets, using only 350 bp of context. Removing the Fourier regularizer sharply reduces this performance.

    Fig. 4: higher correlation than the DART-EVAL DNALMs on both lymphoblastoid cell line (LCL) QTL datasets (350 bp window vs 2,114 bp in DART-EVAL). Exact correlations are not given in the text. · 2.3 ARSENAL improves zero-shot variant effect…; 4.6 Zero-Shot Variant Effect Prediction

  3. ShownReplacing the one-hot input of a ChromBPNet-style model with ARSENAL per-base embeddings improves held-out counts prediction in all five cell lines. In GM12878 models it also improves dsQTL and caQTL variant scoring.

    Fig. 5 (DNase-seq, GM12878, HepG2, K562, IMR90, H1-hESC) and Table 1 (GM12878 models), against ChromBPNet retrained by the authors. Numeric values are not given in the text. · 2.4 ARSENAL embeddings improve supervised…; 4.7 Supervised Model Training; 4.8 Supervised Variant Effect Prediction; Tables

  4. SuggestedAt two example loci (a β-globin enhancer and a chr3 promoter-like cCRE), ARSENAL's likelihood tracks show known motifs, and its nucleotide dependency maps show diagonal blocks aligned with them. The no-prior model, HyenaDNA and Caduceus give weaker or less structured reconstructions.

    Fig. 2, qualitative comparison on two loci: MAFK/NFE motifs at the enhancer; SP1, USF and NRF family motifs at the promoter-like region. · 2.2 ARSENAL improves zero-shot motif discovery; 4.5 Nucleotide Dependency Analysis

  5. SuggestedARSENAL combined with beam search and ChromBPNet oracles produced sequences with graded predicted HepG2 accessibility. It also produced HepG2- versus H1-hESC-specific sets, with differences in motif content that match the design objective. This was evaluated in silico only.

    Fig. 6: separation of predicted counts between the two cell lines. TF-MoDISco found FOX, HNF and C/EBP motifs in HepG2-specific designs, and OCT4 and SOX2 in H1-hESC-specific ones (5 runs each way, 100 sequences per run). · 2.5 ARSENAL can be used for targeted regulatory…; 4.9 Sequence Generation

Methods, data and reproducibility

A masked language model with an 8-block transformer encoder is pretrained on 350 bp ENCODE candidate cis-regulatory elements. Training adds a Fourier-domain loss on the per-base likelihoods, which penalizes energy outside the 6-20 bp motif range. It is evaluated zero-shot, as embeddings for ChromBPNet, and as a generative prior for oracle-guided design.

  • ENCODE candidate cis-regulatory elements (cCREs) for pretraining; split by chromosome (validation chr6, chr21; test chr5, 10, 14, 18, 20, 22) · human (ENCODE cCREs for the human genome) · n = ~2.3 million cCREs
  • K562 PRO-cap promoter peaks for TF-MoDISco · human (K562 cell line) · ENCSR261KBX
  • HepG2 DNase-seq peaks for TF-MoDISco · human (HepG2 cell line) · ENCSR149XIL
  • DART-EVAL zero-shot variant benchmark: Yoruban LCL dsQTLs and African-cohort caQTLs · human
  • DNase-seq data for supervised ChromBPNet-style models (GM12878, HepG2, K562, IMR90, H1-hESC) · human (named human cell lines) · n = five cell lines
Methods & tools
ARSENAL, ChromBPNet, TF-MoDISco, DART-EVAL, HyenaDNA, Caduceus, HOCOMOCO, ENCODE cCREs, PRO-cap
Data & accessions
https://sageb.io/4ZpEnk
Reproducibility gaps
No software versions, containers or hardware/runtime are stated. Models and data are hosted at a single external link.
Limitations (5)
  • ARSENAL is cell-type agnostic and uses a 350 bp window, so it captures local motif grammar but not long-range dependencies such as enhancer-promoter coupling. · stated by the authors
  • DNA language models can learn non-functional patterns such as repeats at high confidence. Sequence-only learning is limited by motif frequency in the training data, and supervised models can better emphasize rare but active motifs. · stated by the authors
  • The generated sequences were evaluated mainly in silico with ChromBPNet oracles and need experimental validation. The best way to handle repeat regions is also left open. · stated by the authors
  • The ablation is confounded: the no-Fourier model excluded all repeat nucleotides from the loss, while the full model excluded only repeat stretches of 30 bp or more. The prior's effect is therefore not isolated. · noted in this summary
  • Supervised variant scoring was tested only with GM12878 models on two LCL QTL datasets. The text gives no numeric effect sizes or variance across training seeds. · noted in this summary

Read this if you want a compact regulatory-DNA language model with a motif-scale Fourier regularizer for zero-shot variant scoring, ChromBPNet inputs, or oracle-guided design.

Citations
0

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISKarollus '24Ji '21Lal '24Fishman '24Salvatore '22Tang '25
64% semantic matchBioinformatics
Alexander Karollus, Johannes Hingerl, Dennis Gankin+3 more·Genome biology·Apr 2, 2024· 57 citations