Fast Loading and Transformation of Large-Volume Bioimaging Data Consistently Approaching the I/O Peak Speed

Abstract

High-resolution large-volume biological imaging techniques are now widely used in biological research, but inefficiencies in data processing and visualization persist due to bottlenecks in loading/saving pipelines and limitations of conventional pyramid formats. To address these issues, we have decoupled data loading process into reading from disk and in-memory decompression, while data saving has been separated into in-memory compression and writing to disk. This approach allows us to dedicate more CPU cores to decompression and compression tasks, optimizing resource allocation and maximizing disk I/O throughput. As a result, both loading and saving processes operate near the disks peak data transfer capacity. Further, we integrated a video stream-inspired format into the pyramid structure, allowing direct access to regions of interest (ROIs) without loading extraneous data into RAM, reducing memory overhead. Compared to the latest methods, our approach improves pyramid data transformation efficiency by at least 6x and dramatically accelerates downstream tasks like visualization and deep learning inference. TeaserNovel bioimaging data processing pipeline achieves near-peak I/O throughput via decoupled architecture and video-formatting.