henhao
Sign in
BioinformaticsbioRxiv
Preprint

Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions

Jiayang CheniD, Zhihang HuiD, Siqi SuniD, Qingxiong TaniD, Yixuan WangiD, Qinze YuiD, Licheng ZongiD, Liang HongiD, Jin XiaoiD, Tao SheniD, Irwin KingiD, Yu LiiD

Abstract

Non-coding RNA structure and function are essential to understanding various biological processes, such as cell signaling, gene expression, and post-transcriptional regulations. These are all among the core problems in the RNA field. With the rapid growth of sequencing technology, we have accumulated a massive amount of unannotated RNA sequences. On the other hand, expensive experimental observatory results in only limited numbers of annotated data and 3D structures. Hence, it is still challenging to design computational methods for predicting their structures and functions. The lack of annotated data and systematic study causes inferior performance. To resolve the issue, we propose a novel RNA foundation model (RNA-FM) to take advantage of all the 23 million non-coding RNA sequences through self-supervised learning. Within this approach, we discover that the pre-trained RNA-FM could infer sequential and evolutionary information of non-coding RNAs without using any labels. Furthermore, we demonstrate RNA-FMs effectiveness by applying it to the downstream secondary/3D structure prediction, SARS-CoV-2 genome structure and evolution prediction, protein-RNA binding preference modeling, and gene expression regulation modeling. The comprehensive experiments show that the proposed method improves the RNA structural and functional modelling results significantly and consistently. Despite only being trained with unlabelled data, RNA-FM can serve as the foundational model for the field.

PDFDOI
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
247
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (7 Aug 2022) · CC BY-NC-ND 4.0

A BERT-style model pretrained on 23.7 million unlabelled ncRNA sequences gives embeddings that improve secondary structure, contact, distance-map and some function predictions over task-specific and MSA-based baselines.

Labelled RNA structure and function data are scarce (the paper notes most methods use at most 30K annotated sequences), and task-specific deep models generalise poorly to unseen RNA types. RNA-FM instead learns from millions of unannotated RNAcentral sequences by self-supervision and is reused across tasks by swapping only the prediction head, which the authors present as the first RNA foundation model of this kind.

Key results

  1. ShownRNA-FM embeddings fed to a simple ResNet head gave the best secondary-structure accuracy on almost all metrics, ahead of SPOT-RNA and UFold on ArchiveII600 and the low-redundancy bpRNA-1m test set TS0.

    Reported margins of 22.8 and 7.5 points over SPOT-RNA and 3.4 and 4.0 points over UFold on ArchiveII600 and TS0 respectively (the introduction gives F1 gains of up to 7.5% and 4% on the low-redundancy set); matched or beat UFold on 85.5% of ArchiveII600 sequences, most clearly above 150 nt. The full comparison table is not in this text. · RNA-FM benefits both RNA secondary structure…; Introduction; RNA second-order structure modeling

  2. ShownFor 3D closeness (contact) prediction, a single ResNet32 on RNA-FM embeddings outperformed RNAcontact's alignment-derived features without needing multiple sequence alignments.

    On TE80, long-range Top-L precision rose 7 points over MSA covariance plus PETfold secondary structure, and another 20 points when the ResNet32 was initialised from bpRNA-1m secondary-structure training; RNA-FM(TL) matched or beat Cov+SS on 77.5% of targets. The introduction states a 30% gain over RNAcontact's 100-model ensemble. · RNA-FM benefits both RNA secondary structure…; Introduction

  3. ShownIn an RNA distance-map regression task defined by the authors, RNA-FM embeddings carry more useful signal than raw sequence or predicted secondary structure.

    Against sequence-only input on TE80: about 39% higher R2 and pixel accuracy, 41% higher Pearson correlation (PMCC) and MSE lower by 0.026 (42%). RNA-FM+Seq beat SS+Seq on 94.2% of targets; adding MSA covariance gave the lowest MSE, 0.0319. · RNA-FM benefits both RNA secondary structure…

  4. ShownGains carry over, more modestly, to function tasks: predicting 5'UTR mean ribosome load and RNA-protein binding.

    Ribosome load: R2 0.875 and MSE 0.247 on the synthetic 5'UTR library, R2 0.816 and MSE 0.264 on 7600 real human 5'UTRs, better than one-hot sequence (baseline values not in this text); adding RNA-FM-predicted structure helped further. On 17 HeLa RBP subsets, RNA-FM plus sequence was best on nearly half and close to PrismNet with in vivo icSHAPE structure. · RNA-FM generalizes to mRNA untranslated region’s…; RNA-FM carries secondary structure information…; Protein-RNA interaction; mRNA untranslated region’s function

  5. SuggestedWithout any labels, the embeddings group ncRNAs by type and recover rough evolutionary orderings for lncRNAs across species and for SARS-CoV-2 variants.

    UMAP of the RNA Atlas (up to 10,000 sequences per type) shows clear type clusters, unlike randomly initialised or one-hot encodings; VIA trajectories on lncRNA embeddings follow the species timeline, and on averaged whole-genome embeddings broadly agree with a FastME phylogeny from Alpha to Omicron. Judged visually; no quantitative score is given. · RNA-FM learns multi-dimension biological…; RNA-FM facilities SARS-CoV-2 genome secondary…; RNA foundation model-generated embeddings…; SARS-CoV-2 virus genome embedding extraction

Methods, data and reproducibility

A 12-block BERT-style transformer (640-dimensional embeddings, 20 attention heads) was pretrained by masked-nucleotide prediction on deduplicated RNAcentral ncRNAs, then used frozen or fine-tuned to feed simple downstream models: a 32-block ResNet for secondary-structure and contact maps, a U-Net for distance maps, PrismNet for protein binding and a convolutional 5'UTR model for ribosome load.

  • RNAcentral100: RNAcentral ncRNAs deduplicated with cd-hit-est at 100% identity (pretraining) · broad range of organisms · n = 23.7 million sequences (from about 27 million across 47 databases)
  • Secondary-structure benchmarks: RNAStralign, ArchiveII (tested on ArchiveII600) and bpRNA-1m (TR0/VL0/TS0 split) · n = 37149; 3975; 13,419 sequences (10,814/1300/1,305)
  • RNAcontact benchmark of non-redundant RNA 3D structures (contact and distance prediction) · n = 221 training (TR221), 80 test (TE80)
  • PDB RNA sequences with 3D structures for the end-to-end 3D model · n = 1036 training; 100 validation/test
  • SARS-CoV-2 regulatory segments and variant genomes from the SARS-CoV-2 Data Hub · SARS-CoV-2 · n = up to 100 genomes per variant
  • HeLa RNA-protein binding data with icSHAPE in vivo structure profiles (PrismNet setup) · human (HeLa cells) · n = 17 RBP subsets
  • Synthetic human 5'UTR library with mean ribosome load from massively parallel reporter assays, plus real human 5'UTRs · human · n = 83,919 synthetic UTRs (7600 held out); 7600 real UTRs
Methods & tools
RNA-FM, cd-hit-est, ResNet32, UFold, SPOT-RNA, RNAcontact, PETfold, PrismNet, 3dRNA, VIA, UMAP, FastME
Assay conditions & compute
Pretraining took one month on eight A100 GPUs with 80 GB memory each; maximum input length 1024 tokens.
Reproducibility gaps
Code, weights and a web server are said to be available, but the links were lost from this text; the SARS-CoV-2 RefSeq accession is also missing, and 3D reconstruction results sit in supplementary tables not included here.
Limitations (6)
  • Improvements on function tasks are smaller than on structure tasks, which the authors attribute to sequence-distribution differences from the ncRNA pretraining set. · stated by the authors
  • The embeddings separate housekeeping from regulatory ncRNAs but struggle to separate long from short ncRNAs, and have difficulty telling lncRNAs of different species apart. · stated by the authors
  • Accuracy on real human 5'UTRs is lower than on the synthetic library because their distributions differ. · stated by the authors
  • Inputs are limited to 1022 nucleotides, so whole viral genomes require windowed embeddings averaged into one vector. · stated by the authors
  • 3D evidence in this text is one example (TS1 RMSD 7.91 vs 25.70 for UFold) plus supplementary tables not included; the evolution analyses are visual trajectory comparisons. · noted in this summary
  • Contact and distance results rest on one 80-RNA test set, and the distance task was defined by the authors, so no external method is benchmarked on it. · noted in this summary

Who should read:Read this if you need general-purpose RNA sequence embeddings for secondary structure, contact or distance prediction, or for RNA function models, without building multiple sequence alignments.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISZhu '25Wang '26Noviello '20Upadhyay '25Booy '22Kagaya '25
78% semantic matchBioinformatics
Heqin Zhu, Ruifeng Li, Feng Zhang+7 more·bioRxiv·Aug 6, 2025· 6 citations