A structured evaluation and benchmarking of Boolean logical modelling tools

Biomolecules form complex interaction systems at different biological levels. The dynamics behind the interactions of biomolecules result in cellular phenotypes and responses. Two typical biological interaction systems are signal transduction and gene regulatory networks. Boolean models characterise these systems components with binary state variables and their interactions with rules governing their influence on each other, without requiring kinetic and abundance information about these components. Tools are available to define, analyse, and manage these models, but there is a lack of an overview and evaluation of such tools quality and functionality. This work aims to facilitate the selection of software tools for Boolean modelling of signal transduction and gene regulatory networks by developing benchmark and evaluation criteria and applying them to the available tools. Twenty Boolean modelling tools were identified, which were assessed on their compliance to ten FAIR4RS (Findability, Accessibility, Interoperability, and Reusability for Research Software) aspects. Additionally, usability was assessed using a set of 25 developed sub-criteria. Furthermore, the functionalities were analysed, validated and the tools abilities to simulate and analyse Boolean models of different sizes were benchmarked. The evaluation demonstrated that, whilst the results produced by the 20 tools were completely consistent, their analytical functionalities and capabilities to simulate larger models varied widely. Size constraints for tools functionalities ranged between 25 and more than 10,000 nodes. Each tool only covered part of the available functionalities, and many exhibited deficiencies in FAIR4RS compliance and usability. Based on our analysis, we see great potential for researchers to improve the usability of their tools along the FAIR4RS aspects in the future, with average tools in our evaluation not yet addressing 40% of the aspects (20% for the best-performing one). The aim is to present a useful overview of usability and functionality of Boolean modelling tools and to show how they can be improved throughout the entire community. Author summaryOur study provides a set of criteria and a benchmarking toolkit for evaluating FAIR4RS and usability aspects as well as functionalities of Boolean modelling tools. We establish an overview of a wide range of Boolean modelling tools for signal transduction and gene regulatory network modelling, which is intended to make it easier for users to choose software according to their needs. This work empowers developers by offering crucial assets: a standardised evaluation criteria set for comparing their tools functionalities, usability, and FAIR4RS compliance throughout the community. Our models and benchmarking assets further enable developers to precisely determine and improve their softwares model size limitations.