LEMMI: A continuous benchmarking platform for metagenomics classifiers

Studies of microbiomes are booming, as well as the diversity of computational tools to make sense out of the sequencing data and the volumes of accumulated microbial genotypes. LEMMI (https://lemmi.ezlab.org) is a novel concept of a benchmarking platform of computational tools for metagenome composition assessments that introduces: a continuous integration of tools, their multi-objective ranking, and an effective distribution through software containers. Here, we detail the workflow and discuss the evaluation of some recently released methods. We see this platform eventually as a community-driven effort: where method developers can showcase novel approaches and get unbiased benchmarks for publications, while users can make informed choices and obtain standardized and easy-to-use tools.