TERSE: Efficient compression of the diffraction data

High-throughput data collection in crystallography poses significant challenges in handling massive amounts of data. Here, we present TERSE, a novel lossless compression algorithm specifically designed for diffraction data. We compare TERSE with the established lossless compression algorithms implemented in gzip, CBF, and HDF5, in terms of compression efficiency and speed, using continuous rotation electron diffraction data of an inorganic compound. Our results show that TERSE outperforms these algorithms by achieving a higher data compression at a speed that is at least an order of magnitude faster. TERSE files are byte-order independent and the algorithm can be readily implemented in hardware. By providing a tailored solution for diffraction data, TERSE facilitates more efficient data analysis and interpretation while mitigating storage and transmission concerns. TERSE C++20 compression/decompression code and an ImageJ/Fiji java plugin for reading TERSE files are open-sourced on GitHub under the permissive MIT license. SynopsisWe present a fast and lossless algorithm for compressing diffraction data, achieving up to 85% reduction in file size while processing up to 2000 512x512 frames per second. This breakthrough in compression technology is a significant step towards more efficient analysis and storage of large diffraction datasets.