Vcflib and tools for processing the VCF variant call format

Erik P. Garrison; Zev N. Kronenberg; Eric T. Dawson; Brent S. Pedersen; Pjotr Prins

doi:10.1101/2021.05.21.445151

bioRxivMay 21, 2021BioinformaticsPreprint

Vcflib and tools for processing the VCF variant call format

Erik P. Garrison iD, Zev N. Kronenberg iD, Eric T. Dawson iD, Brent S. Pedersen iD, Pjotr Prins iD

PDF DOI Code

In-depth full-text digest available · Jump to digest ↓

Abstract

Since its introduction in 2011 the variant call format (VCF) has been widely adopted for processing DNA and RNA variants in practically all population studies -- as well as in somatic and germline mutation studies. VCF can present single nucleotide variants, multi-nucleotide variants, insertions and deletions, and simple structural variants called against a reference genome. Here we present over 125 useful and much used free and open source software tools and libraries, part of vcflib tools and bio-vcf. We also highlight cyvcf2, hts-nim and slivar tools. Application is typically in the comparison, filtering, normalisation, smoothing, annotation, statistics, visualisation and exporting of variants. Our tools run daily and invisibly in pipelines and countless shell scripts. Our tools are part of a wider bioinformatics ecosystem and we consider it very important to make these tools available as free and open source software to all bioinformaticians so they can be deployed through software distributions, such as Debian, GNU Guix and Bioconda. vcflib, for example, was installed over 40,000 times and bio-vcf was installed over 15,000 times through Bioconda by December 2020. We shortly discuss the design of VCF, lessons learnt, and how we can address more complex variation that can not easily be represented by the VCF format. All source code is published under free and open source software licenses and can be downloaded and installed from https://github.com/vcflib. Author summaryMost bioinformatics workflows deal with DNA/RNA variations that are typically represented in the variant call format (VCF) -- a file format that describes mutations (SNP and MNP), insertions and deletions (INDEL) against a reference genome. Here we present a wide range of free and open source software tools that are used in biomedical sequencing workflows around the world today.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (23 May 2021) · CC BY 4.0

A C++ library plus 83 command line utilities and 44 scripts (vcflib) for filtering, transforming and computing population-genetic statistics on VCF files, with companion tools bio-vcf, slivar, cyvcf2 and hts-nim.

VCF is produced by practically every major variant caller, and large sequencing projects need efficient ways to convert, quality-check and compute statistics on it. vcflib had not been published before (yet drew 132 Google Scholar citations in 2020 via its repository), and the paper presents it with the authors' bio-vcf, slivar, cyvcf2 and hts-nim as three styles of VCF processing.

Key results

Shownvcflib provides a C++ API and a large set of small command line programs for reading, filtering, annotating and transforming plain or compressed VCF, meant to be chained by Unix pipes without intermediate files.
83 command line utilities and 44 scripts (over 125 commands in the online documentation), over 30,000 lines of C++; classes vcflib::VariantCallFile and vcflib::Variant with iterators used by every tool, plus a boolean filtering language; tools described include vcfleftalign, vcfclassify, vcf2dag, vcfannotate and vcfglxgt. · Introduction; 2.1 vcflib C++ tools and libraries; Tables
SuggestedGPAT++, bundled in vcflib, computes population-genetic and association statistics directly from VCF, including a Bayesian Fst (bFst) that uses genotype likelihoods supplied by variant callers.
Weir & Cockerham Fst (wcFst), probabilistic pFst, Bayesian bFst with five free parameters (each subpopulation's allele frequency and Fis, total allele frequency, Fst), pVst for SV copy number, Pi and eHH (sequenceDiversity), LD (vcfld), permutation p-values (permuteGPAT++); only a smoothed pFst plot (Fig 2) is shown, with no validation. · 2.1 vcflib C++ tools and libraries; Figure legends
Suggestedbio-vcf (Ruby) and slivar (JavaScript) replace many single-purpose tools with expression languages for filtering and rewriting VCF; bio-vcf parses lazily and runs on multiple cores, and slivar applies expressions across pedigrees and tumor-normal groups.
bio-vcf infers field names such as s.dp and r.tumor.bcount from each file, parses only the fields a query uses, shares RAM across processes via Linux copy-on-write, and emits JSON, RDF, LaTeX, HTML or BED through templates (Fig 1); slivar finds trio de novo variants restricted to gnomad allele frequency < 0.001. No speed benchmarks are given. · 2.2.1 bio-vcf; 2.2.2 Slivar; Figure legends
ShownThe tools are widely installed even though vcflib had never been formally published; the authors credit the continued development of vcflib, bio-vcf and cyvcf2 to the original authors encouraging others to take ownership and release versions, unlike the abandoned pyvcf project.
Bioconda installs by December 2020: vcflib over 40,000 and bio-vcf over 15,000, plus 110,000 Bioconda downloads of freebayes, which incorporates vcflib; Google Scholar lists 132 citations in 2020 to the vcflib GitHub repository. · Introduction; Software development and distribution practices
ClaimedThe authors argue VCF is reaching its limits: it is tied to one reference genome, cannot express nested variation, loses information irrecoverably and has weak metadata; they favour pangenome graphs and, for new formats, JSON, JSON-LD or RDF with ontologies.
Position argument without supporting data: problems cited include reference bias, no representation of superbubbles, ultrabubbles or cacti, header parsing that ignores ontologies, and records that cannot be robustly validated; standardization is at VCFv4.3, yet producing tools take liberties with it. · Discussion; Future work

Methods, data and reproducibility

A software description, not an experiment: vcflib exposes VCF records through C++ classes and iterators used by piped command line tools and the GPAT++ statistics toolkit; bio-vcf (Ruby) and slivar (JavaScript) provide query languages, and cyvcf2 (Python) and hts-nim (Nim) are programming libraries. Features are illustrated with examples; no benchmarks are run.

Methods & tools: vcflib, GPAT++, bio-vcf, slivar, cyvcf2, hts-nim, vcfanno, htslib, vt, freebayes, vcftools, bcftools
Reagents & environment: vcflib and bio-vcf are distributed through Bioconda (install counts reported); the authors stress packaging in Bioconda, Debian and GNU Guix but print no versions, containers or requirement files.
Reproducibility gaps: No repository or documentation URLs are printed (the vcflib GitHub repository and tool websites are mentioned only by name); no version numbers, test data or benchmark scripts are given.

Limitations (6)

VCF depends on a single reference genome, which biases variant calling and misses variation absent from that reference; comparing VCFs across references, or across versions of one reference, is difficult. · stated by the authors
VCF cannot represent complex nested variation (superbubbles, ultrabubbles, cacti), and producing a VCF discards data, so the original information cannot be rebuilt from it. · stated by the authors
Producers take liberties with the VCF 'standard', so parsers must ignore input they do not understand; SV extensions are largely ignored by most processing tools; and robust validation of VCF records is virtually impossible. · stated by the authors
Documentation and regression tests are still being extended; CWL tool descriptions are planned but not yet available. · stated by the authors
No performance benchmarks or head-to-head comparisons with vcftools, bcftools or other parsers are reported, so the speed claims for bio-vcf (lazy parsing, multi-core) and cyvcf2 are unquantified. · noted in this summary
The GPAT++ statistics (wcFst, pFst, bFst, pVst) are not validated here against simulated or known data; the only output shown is one smoothed pFst plot whose underlying dataset is not described. · noted in this summary

Read this if you build or maintain variant-processing pipelines and want an overview of vcflib's command line tools and population-genetic statistics, or of Ruby, JavaScript, Python and Nim alternatives for querying VCF.