VData: Temporally annotated data manipulation and storage

Abstract

BackgroundRecent advances in both single-cell sequencing technologies and gene expression simulation algorithms have led to the production of increasingly large datasets. Larger datasets (tens or hundreds of Gigabytes) can no longer fit on regular computers RAM and thus pose important challenges for storage and manipulation. Existing solutions offer partial solutions but do not explicitly handle the temporal dimension of simulated data and still require large amounts of RAM to run. ResultsVData is a Python extension to the widely used AnnData format that solves these issues by extending 2D dataframes to 3 dimensions (cells, genes and time). VData is built on top of Ch5mpy, a custom built Python library for easily working with hdf5 files and which allows to reduce the memory footprint to the minimum. ConclusionsVData allows to store and manipulate very large datasets of (empirical or simulated) time-stamped data. Since it follows the original Ann-Data format, it is compatible with the scverse tools and AnnData users will find it easy to use.