Deploying genomics workflows on high performance computing (HPC) platforms: storage, memory, and compute considerations

Abstract

Next Generation Sequencing (NGS) workloads largely consist of pipelines of tasks with heterogeneous compute, memory, and storage requirements. Identifying the optimal system configuration has historically required expertise in both system architecture and bioinformatics. This paper outlines infrastructure recommendations for one commonly used genomics workload based on extensive benchmarking and profiling, along with recommendations on how to tune genomics workflows for high performance computing (HPC) infrastructure. The demonstrated methodology and learnings can be extended for other genomics workloads and for other infrastructures such as the cloud.