Sign in
BioinformaticsbioRxiv
Preprint

Byte-exact, format-aware compression for FASTA, FASTQ, BED and VCF files

Michail Patsakis, Alexandros MargarisiD, Theodore Chronopoulos, Ioannis MouratidisiD, Ilias Georgakopoulos‐SoaresiD

Abstract

Motivation Most omics data is stored by compressors that treat it as an unstructured byte stream. Format-aware compressors close that gap but are fragmented across formats and often expect a training pass over the user's own file. A general-purpose lossless compressor that is applicable to the most storage-intense omics data formats while beating the general codecs on ratio and speed would offer significant savings for large omics datasets storage and data retrieval. Results We present NYX, a compressor for FASTA, FASTQ, BED and VCF files. It leverages a reversible, format-specific transform optimised for the OpenZL compression algorithm to reach compression ratios up to 2.3 times those of the strongest general-purpose codecs (zstd -19, xz -9e and 7-Zip) while compressing up to two orders of magnitude faster. On the 1000 Genomes VCF dataset NYX reaches a compression ratio of 468.67x, 2.0 times that of xz -9e while compressing 1.8 times faster. On mammalian FASTA assemblies totalling 9.3 GB, NYX is within 3-4% of NAF, the strongest purpose-built FASTA codec, while compressing 57-73 times faster. Across all 54 inputs no single codec beats NYX on ratio and compression speed together: the tools that achieve higher compression are 1.2 to 73 times slower. NYX provides fully lossless, fast and efficient compression under one easy-to-use tool for the most common omics data types. Availability and implementation Source code, the shipped models, the benchmark harness and the scripts that regenerate every figure and table: https://github.com/Georgakopoulos-Soares-lab/compression

PDFDOICode

02 /Specification & Classification

Click any tag to browse matching papers

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Similarity Map
SelectedSame fieldCross-field
THISKryukov '19Tkachenko '26Nazari '24Roguski '18Lan '23Bonfield '18
71% semantic matchBioinformatics
Kirill A. Kryukov, Mahoko Takahashi Ueda, So Nakagawa+1 more·Bioinformatics·Feb 22, 2019· 58 citations