Galaxy-Kubernetes integration: scaling bioinformatics workflows in the cloud

Pablo A Moreno; Luca Pireddu; Pierrick Roger; Nuwan Goonasekera; Enis Afgan; Marius van den Beek; Sijin He; Anders O. Larsson; Daniel Schober; Christoph Ruttkies; David Johnson; Philippe Rocca‐Serra; Ralf J. M. Weber; Björn Grüening; Reza M. Salek; Namrata Kale; Yasset Pérez‐Riverol; Irene Papatheodorou; Ola Spjuth; Steffen Neumann

doi:10.1101/488643

BioinformaticsbioRxivFeb 12, 2019

Preprint

Galaxy-Kubernetes integration: scaling bioinformatics workflows in the cloud

Pablo A Moreno iD, Luca Pireddu iD, Pierrick Roger iD, Nuwan Goonasekera iD, Enis Afgan iD, Marius van den Beek iD, Sijin He iD, Anders O. Larsson iD, Daniel Schober iD, Christoph Ruttkies iD, David Johnson iD, Philippe Rocca‐Serra iD, Ralf J. M. Weber iD, Björn Grüening, Reza M. Salek iD, Namrata Kale iD, Yasset Pérez‐Riverol iD, Irene Papatheodorou iD, Ola Spjuth iD, Steffen Neumann iD

Abstract

Making reproducible, auditable and scalable data-processing analysis workflows is an important challenge in the field of bioinformatics. Recently, software containers and cloud computing introduced a novel solution to address these challenges. They simplify software installation, management and reproducibility by packaging tools and their dependencies. In this work we implemented a cloud provider agnostic and scalable container orchestration setup for the popular Galaxy workflow environment. This solution enables Galaxy to run on and offload jobs to most cloud providers (e.g. Amazon Web Services, Google Cloud or OpenStack, among others) through the Kubernetes container orchestrator. AvailabilityAll code has been contributed to the Galaxy Project and is available (since Galaxy 17.05) at https://github.com/galaxyproject/ in the galaxy and galaxy-kubernetes repositories. https://public.phenomenal-h2020.eu/ is an example deployment. Suppl. InformationSupplementary Files are available online. Contactpmoreno@ebi.ac.uk, European Molecular Biology Laboratory, EMBL-EBI, Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, UK, Tel: +44-1223-494267, Fax: +44-1223-484696.

PDF DOI Code