Redesigning the Eterna100 RNA design benchmark for the Vienna 2 folding engine

The rational design of RNA is becoming important for rapidly developing technologies in medicine and biochemistry, spurring development of numerous RNA secondary structure design algorithms and benchmarks to evaluate their performance. However, the problem of RNA design is dependent upon the inverse problem of RNA structure prediction through "folding engines" that predict structure from sequence. We hypothesized that differences in RNA folding engines could impact design algorithms, and recruited an online community of RNA design experts to modify the widely used RNA secondary structure design benchmark Eterna100 to address target unsolvability when changing the folding engine used (ViennaRNA 1 updated to 2). We tested this new Eterna100-V2 benchmark with six RNA design algorithms, and found that the performance of RNA design algorithms changed depending on which folding engine was used in training or evaluation. We further explored algorithms reported to have solved the original Eterna100 and found that incorrect folding engine settings were used in their prior evaluation. As a result, currently, both the original Eterna100-V1 and the modernized Eterna100-V2 benchmark remain unsolved with their intended folding engine settings. This work demonstrates that the design difficulty of RNA structures is intrinsically linked to thermodynamic models and suggests that future RNA design algorithms that can adapt to updated thermodynamic models may result in improved performance. Eterna100-V1 and Eterna100-V2 benchmarks and example solutions are freely available at https://github.com/eternagame/eterna100-benchmarking.