henhaobio
Sign in

bioRxivBioinformaticsPreprint

The ENCODE Uniform Analysis Pipelines

Benjamin C. HitziD, Jin-Wook LeeiD, Otto JolankiiD, Meenakshi S. KagdaiD, Keenan GrahamiD, Paul SudiD, Idan GabdankiD, J. Seth StrattaniD, Cricket Alicia SloaniD, Timothy R. DreszeriD, Laurence D. RoweiD, Nikhil R. PodduturiiD, Venkat S. MalladiiD, Esther T. ChaniD, Jean M. DavidsoniD, Marcus HoiD, Stuart R. MiyasatoiD, Matt SimisoniD, Forrest Y. TanakaiD, Yunhai LuoiD, Ian WhalingiD, Khine Zin LiniD, Jennifer JouiD, Eurie L. HongiD, Brian T. LeeiD, , J. Michael CherryiD

PDFDOI
In-depth full-text digest available · Jump to digest ↓

Abstract

The Encyclopedia of DNA elements (ENCODE) project is a collaborative effort to create a comprehensive catalog of functional elements in the human genome. The current database comprises more than 19000 functional genomics experiments across more than 1000 cell lines and tissues using a wide array of experimental techniques to study the chromatin structure, regulatory and transcriptional landscape of the Homo sapiens and Mus musculus genomes. All experimental data, metadata, and associated computational analyses created by the ENCODE consortium are submitted to the Data Coordination Center (DCC) for validation, tracking, storage, and distribution to community resources and the scientific community. The ENCODE project has engineered and distributed uniform processing pipelines in order to promote data provenance and reproducibility as well as allow interoperability between genomic resources and other consortia. All data files, reference genome versions, software versions, and parameters used by the pipelines are captured and available via the ENCODE Portal. The pipeline code, developed using Docker and Workflow Description Language (WDL; https://openwdl.org/) is publicly available in GitHub, with images available on Dockerhub (https://hub.docker.com), enabling access to a diverse range of biomedical researchers. ENCODE pipelines maintained and used by the DCC can be installed to run on personal computers, local HPC clusters, or in cloud computing environments via Cromwell. Access to the pipelines and data via the cloud allows small labs the ability to use the data or software without access to institutional compute clusters. Standardization of the computational methodologies for analysis and quality control leads to comparable results from different ENCODE collections - a prerequisite for successful integrative analyses. Database URL: https://www.encodeproject.org/

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 1 (6 Apr 2023) · CC BY 4.0

ENCODE's containerised Workflow Description Language pipelines for ChIP-seq, ATAC-seq, DNase-seq, RNA-seq variants, Hi-C and bisulfite sequencing run on cloud or local clusters, with documented compute costs and provenance.

Pipelines for the same assay often differ in software, parameters and significance rules, so their outputs cannot always be compared, and running a genomics pipeline locally has demanded considerable expertise. The paper describes how ENCODE's Data Coordination Center (DCC) packaged uniform processing for about 20 assays into tested Docker and WDL workflows run through Cromwell, with metadata recording each file's inputs, software versions and QC.

Key results

  1. ShownENCODE runs one uniform pipeline family per assay type, covering ChIP-seq (transcription factor, histone, MINT-ChIP), ATAC-seq, DNase-seq, five RNA-seq pipelines, Hi-C and whole-genome bisulfite sequencing (WGBS), and used them on over 14,000 datasets in its final phase.

    Final phase: 14,000+ datasets (at least 40,000 fastqs) across about 20 assays; over 5800 ChIP-seq assays plus over 1600 controls pass through one ChIP-seq pipeline. RNA-seq pipelines: standard (>200bp), short (<200bp), RAMPAGE/CAGE, long-read and microRNA. The database holds more than 19000 experiments across more than 1000 cell lines and tissues. · Abstract; Introduction; Pipeline Infrastructure (CAPER/CROO); The ENCODE ChIP-seq Pipelines

  2. ShownThe ChIP-seq, DNase-seq, bulk RNA-seq and microRNA-seq pipelines cost at most 8 CPU hours per million reads, with peak memory from 18GB (ChIP-seq) to 120GB (bulk RNA-seq).

    Per million reads: ChIP-seq 1-8 CPU h (average 2) and up to 18GB RAM (average 12 GB); DNase-seq 1.3 CPU h, maximum 32GB; bulk RNA-seq about 1 CPU hr/4GB on Google Cloud, maximum 120GB; microRNA-seq about 0.5 CPU h, maximum 60GB. · The ENCODE ChIP-seq Pipelines; The ENCODE RNA-seq Pipelines; micro-RNA; The ENCODE DNase-Seq Pipeline

  3. ShownHi-C and long-read RNA-seq are the most demanding: a Hi-C run takes about 60 hours of wall time with some steps needing 512 GB RAM, and long-read jobs need about 100 CPU hours per million reads.

    Hi-C: about 1.5 CPU hours/million reads; intact Hi-C reaches 200 billion reads; CPU time led by conversion to merged_nodups (24%) and chimeric-read handling (15%). Long-read RNA (0.5M-3.5M reads per experiment): about 100 CPU hours per 1 million reads and 120GB RAM. WGBS: about 0.02 h wall time per million reads, with mapping (70% of time) on 16 CPUs and 128 GB RAM. · long read RNA; QC metrics; The ENCODE Hi-C Pipeline

  4. ClaimedThe same WDL code is meant to run on Google Cloud, AWS or local clusters: CAPER wraps Cromwell for submission and cross-platform file transfer, Singularity stands in for Docker on HPC schedulers, and several pipelines are also offered via Dockstore, Truwl and Seven Bridges.

    CAPER has SGE, SLURM, PBS and LSF back ends that use Singularity, since most HPCs block Docker; Singularity images and a Conda installer exist for ChIP and ATAC. ChIP-seq, ATAC-seq, RNA-seq, long-read RNA-seq, microRNA-seq, WGBS and Hi-C are on Dockstore; five pipelines are on Truwl and two on Seven Bridges. Unit, step-wise and end-to-end tests run on circle-ci. · Introduction; Pipeline Infrastructure (CAPER/CROO); Using or Installing the ENCODE Pipelines

  5. ClaimedEach processed file on the ENCODE portal is traced to its raw data, software versions and QC metrics, QC thresholds set yellow, orange or red audit badges, and pipeline steps are built to give identical outputs from identical inputs.

    Pipelines, steps, runs, software versions and quality metrics are JSON-SCHEMA objects in encodeD/SnoVault, filled by the accession package from Cromwell metadata; files are linked through a derived_from property. ChIP-seq QC includes FRIP, NSC/RSC, Jensen Shannon distance, PBC and NRF; idempotence is judged by equal md5 checksums of outputs. No QC values are given in this text. · Software and Pipeline Metadata and Provenance; The ENCODE ChIP-seq Pipelines; ENCODE Standards; Discussion

Methods, data and reproducibility

Each pipeline is a chain of containerised steps (mapping, filtering, peak calling or quantification, QC) written in WDL and executed by Cromwell; the CAPER wrapper handles submission and file transfer across cloud and HPC back ends, CROO organises outputs for outside users, and the accession package writes outputs, versions and QC into the ENCODE portal's metadata.

  • ENCODE functional genomics experiments (mostly ChIP-seq, RNA-seq and DNase-seq) across cell lines and tissues · human and mouse · n = over 80,000 fastq files across over 17,000 experiments
  • ChIP-seq experiments, including multiplexed MINT-ChIP histone assays, with matched controls · human and mouse · n = over 5800 assays plus over 1600 controls
  • Long-read RNA-seq datasets from PacBio and Oxford Nanopore · human and mouse · n = approximately 200 datasets
  • microRNA-seq datasets from ENCODE phases 3 and 4 and REMC · n = ∼400 datasets
  • RAMPAGE and CAGE experiments from ENCODE phases 2 and 3 and GGR · n = 289 experiments
Methods & tools
WDL and Cromwell, Docker and Singularity, CAPER, CROO, accession (Python package), bowtie2, SPP, MACS2 and IDR (ChIP-seq, ATAC-seq), STAR 2.5.1b, RSEM and Kallisto (RNA-seq), TALON (long-read RNA-seq), gemBS (WGBS), BWA and hotspot2 (DNase-seq), Juicer, HiCCUPS and 3D-DNA (Hi-C), encodeD / SnoVault
Reagents & environment
Docker images on DockerHub, with a Dockerfile and WDL in each GitHub repository; Singularity images and a Conda installer for ChIP and ATAC; GRCh38 and mm10 genomes with GENCODE V29 and M21; software versions recorded on the ENCODE portal.
Assay conditions & compute
Per million reads: ChIP-seq 1-8 CPU h and up to 18GB RAM; bulk RNA-seq about 1 CPU hr/4GB (max 120GB); long-read RNA about 100 CPU h and 120GB; a Hi-C run about 60 h wall time with some steps at 512 GB RAM.
Reproducibility gaps
The GitHub, DockerHub, Dockstore, portal and reference-file links were lost in extraction; the RAMPAGE/CAGE pipeline has not been ported to WDL/Cromwell; full QC metric reports are in Supplementary tables 1-6, not in this text.
Limitations (6)
  • The RAMPAGE and CAGE pipeline has not been adapted to run in the WDL/Cromwell cloud system. · stated by the authors
  • Large runs over heterogeneous inputs keep exposing bugs; users should expect to re-run jobs that exceed resources or need new parameters, and deeper sequencing will cause further failures. · stated by the authors
  • Hi-C diploidification neither identifies parental haplotypes nor phases across chromosomes, so pseudohaplotype labels are arbitrary for each chromosome. · stated by the authors
  • Most HPC clusters do not allow Docker, so cluster users depend on Singularity, and Singularity images plus a Conda installer are offered only for some workflows (ChIP and ATAC). · stated by the authors
  • No comparison of outputs or accuracy against alternative pipelines is reported, so the tool choices are supported by description rather than benchmarks. · noted in this summary
  • Several pipelines (TF ChIP-seq, WGBS, long-read RNA, DNase footprinting) defer method details and validation to other publications, some still in preparation, so this text alone does not document them fully. · noted in this summary

Read this if you want to process your own ChIP-seq, ATAC-seq, RNA-seq, DNase-seq, Hi-C or WGBS data the way ENCODE does, or need to know how ENCODE's processed files and QC flags were produced.

Citations
149

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISHitz '16Ruffier '17Hutton '25Hung '22Fisch '15Busby '15
62% semantic matchBioinformatics
Benjamin Hitz, Laurence D Rowe, Nikhil Podduturi+18 more·bioRxiv·Jul 29, 2016