GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models

Key results

1. 01

   SuggestedModel code and evaluation are kept apart. A task exposes its attributes, optional fine-tuning data and validation, cross-validation and test entry points. A model implements whichever standard inference methods it supports, and each task then computes only the metrics those outputs allow.

   Task API: get_task_attributes, get_finetune_dataset, eval_validation_set, eval_cross_validation_fold, eval_test_set. Optional inference methods: single-sequence or reference/variant pair classification and sequence-to-sequence outputs (token embeddings, per-token probabilities, one sequence embedding), with automatic position mapping where tokenization shifts alignment. · B.1. Task API; Single-Sequence Classification Inference; Variant-Reference Pair Classification Inference; Sequence-to-Sequence Inference

2. 02

   ShownThe bundled tasks, drawn from established benchmarks, cover supervised sequence classification, supervised variant classification and zero-shot variant scoring. Input windows run from 100 bp to 160,000 bp.

   3 GUE tasks (promoter 300 bp, three-class splice site 400 bp, transcription factor binding 100 bp); six VariantBenchmarks tasks at 1,024 bp; Long Range Benchmark (LRB) causal expression QTL at 160,000 bp, tissue as conditional input; zero-shot BEND (512 bp), TraitGym (4,096 bp), Song Lab ClinVar (5,994 bp), LRB OMIM (12,000 bp in the main text; the appendix says up to 160,000 bp), BRCA1 (8192 bp). · Supervised single-sequence tasks; Supervised variant effect tasks; Zero-shot variant effect tasks; Long Range Benchmark Pathogenic OMIM Prediction

3. 03

   SuggestedFive pretrained genomic foundation models, from about 7M to 1B parameters, were run across the whole task suite through one interface. Their per-task scores are in appendix Tables 1 to 5, whose values are not included in the text.

   DNA-BERT (110M parameters), DNABERT-2 (~117M), NTv3 (~8M) and Caduceus-Ps (~7M), from Hugging Face, were fully fine-tuned for 3 epochs on supervised tasks. Evo 2 (1B, via NVIDIA BioNeMo, FP16 on one H100) was evaluated by linear probing with frozen weights. · Case Study; A. Model Sourcing and Implementation; B. Fine-Tuning and Training Protocols; E. Hardware Configuration

4. 04

   ClaimedEvery task implementation is said to have been checked against its original benchmark code or published results. No agreement figures are given, and several tasks deliberately depart from the original evaluation protocol.

   Departures: the VariantBenchmarks coding and non-coding pathogenicity tasks use a fixed held-out fold (chromosomes 21, 22 and X) instead of the original cross-validation; TraitGym drops reverse-complement averaging and repeated bootstrapping. · Zero-shot general indel tasks; VariantBenchmarks Coding Pathogenicity; VariantBenchmarks Non-Coding Pathogenicity; TraitGym

5. 05

   SuggestedA new open ClinVar indel pathogenicity task scores insertions and deletions zero-shot on reference and alternate sequences of unequal length. It uses only sequence-level likelihood and embedding scores, since per-position comparisons are not defined for length-changing variants.

   Built on BaseGFMZeroShotGeneralIndelTask. Variant-position cosine similarity and the masked-token log-likelihood ratio (LLR) are excluded (SNV cosine similarity is marked NA for indel tasks in Table 3). The text gives no variant count or construction details for the new set. · B.2. Tasks Design; Zero-shot general indel tasks; ClinVar Indel Pathogenicity; Tables