HydraRNA: a hybrid architecture based full-length RNA language model

Abstract

RNA, an essential component of the central dogma of molecular biology, plays versatile roles in all cellular processes. RNA large language models (LLMs) are emerging as powerful methods in RNA research to decipher its intricate network of function and regulation. However, previous RNA LLMs were based on the transformer model and pre-trained on short segment of non-coding RNAs, which limits their general usability. Here we present the first full-length RNA foundation model, HydraRNA, which is based on a hybrid architecture of bidirectional state space model and multi-head attention mechanism, and is pre-trained on a large amount of both protein-coding and non-coding RNAs. Despite being pre-trained with the fewest parameters and the least GPU resources, HydraRNA learns better RNA representations and outperforms the existing foundation models on a variety of downstream tasks, including RNA classification, prediction of RNA secondary structure, RBP binding sites, mRNA stability and translation efficiency. Furthermore, HydraRNA can accurately predict the effect of mutations and estimate the relative contributions of different mRNA regions to the RNA stability and translation. We anticipate that HydraRNA will enable dissecting the diverse properties of RNA, accelerating the research of RNA regulation and facilitating the optimal design of RNA therapeutics.