henhaobio
Sign in

bioRxivBioinformaticsPreprint

BGEN: a binary file format for imputed genotype and haplotype data

Gavin BandiD, Jonathan MarchiniiD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

The impact of modern technology on genetic epidemiology has been significant, with studies comprising millions of individuals assessed at tens of millions of genetic variants now becoming common. Studies on this scale provide logistical and analytic challenges starting with the issue of efficiently storing, transmitting, and accessing underlying data. Here we present a binary file format (the BGEN format) that can store both directly-typed and statistically imputed genotype data, and achieves substantial space savings by data compression and the use of an efficient representation for probabilities. We investigate the properties of this format using imputed data from the UK BiLEVE study, demonstrating both storage efficiency, and fast data loading performance on the order of hundreds of millions of imputed genotypes per second. To make using BGEN as easy as possible, we provide a detailed specification and a freely available reference implementation, and we leverage this by developing additional tools including an indexing tool (bgenix) and an R package (rbgen) that permits loading of BGEN-encoded data into the R statistical programming environment. The UK Biobank is one of a number of projects that have used BGEN for release of imputed data, and we expect the format to continue to be widely implemented and used.

In-depth summary

Claude Opus 5.5 · xhigh effort · 7 Nov 2026 · from the full text of version 2 (2 May 2018) · CC BY 4.0

Encoding imputed genotype probabilities at 8 bits in BGEN gave compact files, ~240 million probabilities parsed per second and association results near-identical to 20-bit data (UK BiLEVE chromosome 22).

With cohorts such as UK Biobank (almost 500,000 individuals, over 80 million variants), first-generation GWAS formats no longer cope, and the linkage-disequilibrium-based compression methods the authors cite do not extend to imputed genotypes. BGEN keeps imputation uncertainty using a fixed-bit probability encoding whose values always sum to one, compresses each variant separately and leaves variant metadata uncompressed for indexing; it was adopted for UK Biobank's imputed data release.

Key results

  1. ShownAt precision comparable to the 3-decimal source, BGEN files are smaller than gzipped GEN or PLINK binary (BED) files. Lower bit depths shrink them further, zstandard produced smaller files than zlib in every comparison, and file size grows roughly linearly with sample count.

    Chromosome 22, 49,458 UK BiLEVE samples: 12-bit BGEN 2.8-3.5Gb depending on compression vs 4.3Gb for zlib level 9 GEN and 6.2Gb for uncompressed PLINK BED; 20-bit BGEN 3.7-4.3Gb; 8-bit 2.3-2.7Gb; 4-bit 1.3-1.6Gb. Size scaled roughly linearly with samples (limiting slope approximately 0.05Mb per sample, 8-bit zstandard). · File size; Partioning and scaling of storage size; Performance comparison

  2. ShownBinary formats were read far faster than the text source. 8- and 16-bit BGEN were the quickest bit depths to parse into memory, and zstandard-compressed BGEN files were the fastest of all formats tested.

    Single-thread QCTOOL read of all genotypes into floating point: GEN source 5,081s; 8-bit BGEN 328s (~240 million genotype probabilities per second); 16-bit BGEN 338s. Zstandard files took around 60-80% of the read time of zlib-compressed files. · File reading performance; Performance comparison

  3. ShownWorking directly on the stored bits, with a lookup table that maps each individual's 16 encoded bits to a dosage, gave the same results as converting to floating point but ran substantially faster.

    Expected allele frequency for all 531,412 variants from 8-bit zstandard BGEN: approximately 57s (~460 million genotypes per second) with the lookup table vs around 200s converting each probability to floating point; the two programs gave identical output. · File reading performance

  4. SuggestedEncodings of 8 bits or more barely change association results compared with 20-bit data, whereas 4 bits produce visible discrepancies. Lower precision also pushes rarer-allele frequency estimates down for poorly imputed variants, an effect almost absent from b=8 upward.

    5 simulated case/control and 100 simulated quantitative traits (SNPTEST, BGENIE): b>=8 max z-score difference 0.11 and max -log10 P difference 0.09 vs b=20 (0.02 and 0.03 for info > 0.5, expected minor allele frequency > 0.5%); b=4 gave 1.6 and 1.1 (0.4 and 0.4 after the same filter). · Downstream effects of lowered precision; Downstream effect of loss of precision

  5. ShownBecause variant identifiers are stored uncompressed, BGEN can list variants and build an index without decompressing any genotypes.

    Listing chromosome, position, ID and alleles for 531,412 variants took approximately 5 seconds from BGEN vs >5 minutes for compressed GEN or VCF (zcat and cut) or BCF (bcftools); essentially instantaneous for the PLINK .bim file. Building a bgenix (sqlite-based) index took approximately 75s. · Listing variants and indexing; Indexing

Methods, data and reproducibility

Defines a binary layout (header, optional sample identifiers, one block per variant) that rounds each sample's genotype or haplotype probabilities to the nearest simplex point expressible in b bits (error at most 1/(2^b-1)) and compresses each variant with zlib or zstandard; benchmarked on UK BiLEVE chromosome 22 against GEN, PLINK BED, VCF and BCF.

  • UK BiLEVE chromosome 22 genotypes imputed into the 1000G Phase 3 panel (9,737 typed, 521,675 imputed variants); source supplied as gzipped GEN at 3 decimal places · human (imputed against the 1000G Phase 3 reference panel) · n = 49,458 samples
  • Random sample subsets of the 8-bit BGEN data, used to measure how file size scales with sample count · human (subsets of the same UK BiLEVE data) · n = 10 to 31,623 samples
  • Simulated phenotypes on the chromosome 22 genotypes: binary traits with case fractions of 10%, 25%, 50%, 75% and 90%, and quantitative traits each with one associated variant (effect size SD 0.04) · n = 5 case/control and 100 quantitative traits
Methods & tools
BGEN C++ reference implementation, bgenix, rbgen, QCTOOL, SNPTEST, BGENIE, PLINK, bcftools, zlib, zstandard, sqlite3, 1000G Phase 3 reference panel
Software & environment
Benchmark code compiled with gcc 5.4.0 with the avx, sse2 and ssse3 instruction sets enabled; no versions, containers or requirement files are given for the BGEN library, bgenix or rbgen.
Assay conditions & compute
One Wellcome Centre for Human Genetics cluster node (Intel Xeon CPU at 2.6Ghz) with data on the cluster storage filesystem; all timings are single-threaded, and each reported time is the best of three runs.
Reproducibility gaps
The UK BiLEVE test data are not shared and no access route is given; the benchmark and simulation scripts (including the lookup-table allele-frequency program) are not linked.
Limitations (6)
  • Only genotypes and genotype probabilities can be stored. Unlike VCF or BCF, BGEN cannot hold sequence read counts or genotype likelihoods, which the authors say may limit its use in some settings. · stated by the authors
  • Each variant is compressed independently. The authors expect haplotype-sharing approaches such as PBWT to compress considerably better and possibly read faster, though those methods do not yet handle imputed genotypes. · stated by the authors
  • Comparator formats were read through QCTOOL's own implementations, which may not be the fastest; PLINK's native BED reader is likely faster than the one benchmarked here. · stated by the authors
  • Ploidy must be known in advance. Phased data carry no phase-block information: haplotypes are assumed to be globally phased within each chromosome, unlike in VCF. · stated by the authors
  • All benchmarks use one cohort and one chromosome (chromosome 22, 49,458 samples) on a single thread. File-size scaling was measured only up to 31,623 samples, not at the million-sample scale that motivates the format. · noted in this summary
  • Precision effects rest on simulated phenotypes (binary traits drawn without genetic signal, one causal variant per quantitative trait). They are judged against the 20-bit or 16-bit BGEN encodings rather than the original 3-decimal source probabilities. · noted in this summary

Read this if you distribute or analyse imputed genotype probabilities at biobank scale and need to pick a bit depth, compression library or indexing approach for BGEN files, or want to write your own BGEN reader.

Citations
64

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISDeHaas '24DeHaas '24Bycroft '17Harmanci '21Chen '21Chu '21
62% semantic matchBioinformatics
Drew DeHaas, Xinzhu Wei·Bioinformatics Advances·Dec 26, 2024· 1 citations