Characterising the performance of an antibiotic resistance prediction tool, gnomonicus, using a diverse testset of 2,663 Mycobacterium tuberculosis samples

Abstract

Tuberculosis remains a global health problem. Making it easier and quicker to identify which antibiotics an infection is likely to be susceptible to will be a key part of the solution. Whilst whole-genome sequencing offers many advantages, the processing of the genetic reads to produce the relevant public health and clinical information is, surprisingly, often the responsibility of the end user which inhibits uptake. Here we characterise how well a freely-available tool we have developed, gnomonicus, predicts the antibiotic resistance profile of a sample (given its variant call file) using our implementation of the second edition of the WHO catalogue of resistance-associated variants (WHOv2). To facilitate this, we have constructed a Diverse Testset of 2,663 publicly-available M. tuberculosis samples which have both genetic and drug susceptibility testing (DST) data. We have chosen to apply the catalogue such that our tool will return a result of (i) Fail if there are insufficient reads at a genetic locus associated with resistance, (ii) Unknown if a genetic variant in a resistance gene not listed in the catalogue is encountered and (iii) Resistant if three or more short-reads support the presence of a resistance-associated variant. The last step increases the sensitivity for all 15 antibiotics but only reaches significance in a few in our testset. Comparing our results to those of TB-Profiler, an existing tool, highlights the different design choices and demonstrates the performance of both tools on our Diverse Testset is comparable. By only considering high confidence DST results we show that gnomonicus, in combination with our translation of WHOv2, achieves sensitivities and specificities in excess of 95% for both isoniazid and rifampicin. Impact StatementWhole genome sequencing clinical samples taken from patients with tuberculosis is a potentially fast and accurate method for determining to which antibiotics the infection will be susceptible. Two barriers need to be overcome; the first, which is knowing which mutations are associated with resistance (or not) to a range of antibiotics is well on the way to be solved thanks to the efforts of the World Health Organization (WHO) who have published extensive catalogues containing lists of such mutations. The second barrier is that the processing of the raw genetic files remains a largely manual process overseen by bioinformaticians. Here we describe gnomonicus, our open-source AMR prediction tool, and report the performance of our translation of the second edition of the WHO catalogue using a carefully designed publicly-available dataset of 2,663 M. tuberculosis samples. We hope that not only will this tool be useful but also that this dataset will be used by other researchers to facilitate comparisons between pipelines, approaches and tools. Data SummaryThe attendant GitHub repository1 allows gnomonicus to be rerun on all 2,663 samples in the Diverse Testset; it therefore includes instructions, the necessary input files (all the variant call files, the version of the WHOv2 catalogue and a link to the H37Rv GenBank file used in this study) and also the output JSON files. The ENA accession numbers for all 2,663 samples (including a bash script to download them) and their corresponding phenotypic drug susceptibility testing results are included with the intention that people can either reproduce our results, or use the same dataset for other analyses. The JSON files output by TB-Profiler have also been added for comparison. The repository contains a series of Juypter notebooks containing Python3 code that allows the user to discover and parse the output JSON files from either tool and save the results as data tables. Other notebooks allow the user to reproduce all the analysis underlying this work, including reproducing the figures and many of the tables.