fastq-pair: efficient synchronization of paired-end fastq files

John A. Edwards; Robert Alan Edwards

doi:10.1101/552885

bioRxivFeb 19, 2019BioinformaticsPreprint

fastq-pair: efficient synchronization of paired-end fastq files

John A. Edwards, Robert Alan Edwards iD

PDF DOI Code

In-depth full-text digest available · Jump to digest ↓

Abstract

Paired end DNA sequencing provides additional information about the sequence data that is used in sequence assembly, mapping, and other downstream bioinformatics analysis. Paired end reads are usually provided as two fastq-format files, with each file representing one end of the read. Many commonly used downstream tools require that the sequence reads appear in each file in the same order, and reads that do not have a pair in the corresponding file are placed in a separate file of singletons. Although most sequencing instruments capable of generating paired end reads produce files where each read has a corresponding mate, many downstream bioinformatics manipulations break the one-to-one correspondence between reads, and paired-end sequence files loose synchronicity, and contain either unordered sequences or sequences in one or other file without a mate. Trivial solutions to this problem require reading one or both of the DNA sequence files into memory but quickly become limited by computational resources for moderate to large sized sequence files that are common nowadays. Here, we introduce a fast and memory efficient solution, written in C for portability, that synchronizes paired-end fastq files for subsequent analysis and places unmatched reads into singleton files. Fastq-pair is freely available from https://github.com/linsalrob/fastq-pair and is released under the MIT license.

In-depth summary

Claude Opus 5.5 · xhigh effort · 7 Nov 2026 · from the full text of version 1 (19 Feb 2019) · CC BY 4.0

fastq-pair, a C tool, re-pairs paired-end fastq files by hashing read identifiers and file offsets, writing matched pairs plus singleton files with linear time and memory scaling.

Tools such as pear, spades and bowtie2 need both paired-end fastq files to hold the same reads in the same order, yet fastq-dump and several trimmers can break this, and simple fixes that load files into memory run out of resources on files of tens of gigabytes. fastq-pair indexes only identifiers and file positions from the first file rather than whole records, which the authors say greatly reduces memory compared with such approaches.

Key results

ShownRun time and peak memory grow linearly with input size across hundreds of real fastq files.
Fig. 1 profiles 422 fastq files from the Sequence Read Archive (SRA), ranging from 66,854 to 127,235,892 sequences. Time (user plus system seconds) and maximum resident memory (MB) were measured with unix time. The authors report linear complexity for both. The paper benchmarks no competing tool or baseline. · fastq-pair; Figure legends
ClaimedMemory savings come from storing only each first-file identifier and its file position, not full sequence and quality records, so matching reads are fetched from disk by random access.
This is stated in the method description. The paper reports no memory measurement for a naive in-memory approach or another pairing tool to compare against. · Abstract; fastq-pair
ClaimedThe output is four files: two synchronized files (input name with .paired.fq added) and one singleton file per input (.single.fq). Singletons are kept in separate files so per-input counts are immediate and file names stay unique for parallel jobs.
This is a design description. Second-file reads go to the paired output if found in the hash, otherwise to that file's singleton file. First-file reads never matched are written as singletons at the end. · fastq-pair
ClaimedThe user can change the hash table size. A larger table uses more memory and takes longer to set up but speeds up lookups. The default is meant to spread entries evenly and keep scaling linear on large inputs.
The default table size is 100,003. The trade-offs are described only in words; no timings across table sizes are reported. · fastq-pair

Methods, data and reproducibility

A C program builds a hash of sequence identifiers and file positions from the first fastq file. It then streams the second file, writing each read either as a matched pair (retrieving the mate by random access) or as a singleton, and finally writes unseen first-file reads as singletons. Scaling was profiled on 422 SRA fastq files with unix time.

Fastq files from the Sequence Read Archive (SRA), 66,854 to 127,235,892 sequences each, used to profile time and memory · n = 422 fastq files

Methods & tools: fastq-pair, unix time, Sequence Read Archive (SRA)
Code: https://github.com/linsalrob/fastq-pair
Reproducibility gaps: The 422 SRA files used for profiling are not identified by accession. No hardware, software version or build details are given. Fig. 1 values appear only as a plot legend in the text.

Limitations (4)

It cannot process compressed fastq files because it depends on random access to the file stream. Decompressing to memory or a temporary file would undo the memory savings. · stated by the authors
It assumes unwrapped fastq records of exactly four lines each (identifier, sequence, + spacer, quality). · stated by the authors
No benchmark against other pairing tools or a naive in-memory method, so the claimed memory and speed gains are not quantified against an alternative. · noted in this summary
The scaling evaluation reports no hardware and does not list the SRA files used, so the measurements cannot be reproduced exactly. · noted in this summary

Read this if you need to re-pair large uncompressed paired-end fastq files after SRA download or trimming, before assembly or mapping, and want a low-memory command-line utility.