NanoPyx: super-fast bioimage analysis powered by adaptive machine learning

To overcome the challenges posed by large and complex microscopy datasets, we have developed NanoPyx, an adaptive bioimage analysis framework designed for high-speed processing. At the core of NanoPyx is the Liquid Engine, an agent-based machine-learning system that predicts acceleration strategies for image analysis tasks. Unlike traditional single-algorithm methods, the Liquid Engine generates multiple CPU and GPU code variations using a meta-programming system, creating a competitive environment where different algorithms are benchmarked against each other to achieve optimal performance under the user"s computational environment. In initial experiments focusing on super-resolution analysis methods, the Liquid Engine demonstrated an over 10-fold computational speed improvement by accurately predicting the ideal scenarios to switch between algorithmic implementations. NanoPyx is accessible to users through a Python library, code-free Jupyter notebooks, and a napari plugin, making it suitable for individuals regardless of their coding proficiency. Furthermore, the optimisation principles embodied by the Liquid Engine have broader implications, extending their applicability to various high-performance computing fields.