Unlocking plant health survey data: an approach to quantify the sensitivity and specificity of visual inspections

Abstract

Invasive plant pests and pathogens cause significant environmental and economic damage. Visual inspection remains a central tenet of plant health surveys, but its sensitivity (probability of correctly identifying the presence of a pest) and specificity (probability of correctly identifying the absence of a pest) is usually ignored. These parameters facilitate calculation of surveillance metrics which are critical for effective contingency planning and outbreak management. To address this, twenty-three citizen scientist surveyors assessed up to 175 oak trees for three symptoms of acute oak decline. The same trees were also assessed by an expert who has monitored these trees annually for over a decade. The sensitivity and specificity of surveyors was calculated using the expert data as the gold-standard. The utility of a workflow utilising Bayesian modelling was then examined using simulated data to estimate these parameters in the absence of a rarely available gold-standard dataset. There was large variation in sensitivity and specificity between surveyors and symptoms, although the sensitivity was positively related to the number of symptoms on a tree. By leveraging surveyor observations of two symptoms from a minimum of 80 trees on two sites, with knowledge of whether a site has higher ([~]0.6) or lower ([~]0.3) true disease prevalence we show that sensitivity and specificity can be estimated without gold-standard data. We highlight that sensitivity and specificity will depend on the symptoms of a pest or disease, the individual surveyor, and the survey protocol. This has consequences for how surveys are designed to detect and monitor outbreaks, as well as the interpretation of survey data that is used to inform outbreak management. Author summaryThe increasing occurrence of emerging plant pests and diseases is affecting both agricultural and natural ecosystems. Effective management and control of such pests and diseases is much easier when they are detected early. Currently, visual surveys underpin plant health surveillance, but basic metrics of the reliability of visual detection such as the sensitivity (probability of correctly identifying a positive) and specificity (probability of correctly identifying a negative) are not routinely quantified. In this study, we first quantify the sensitivity and specificity of 23 trained citizen scientist surveyors at detecting three symptoms of acute oak decline, by comparing their symptom classifications against a dataset from an expert who has conducted long-term monitoring of these trees. We demonstrate how individuals vary greatly in their ability to detect symptoms, and how different symptoms are associated with different detection error. Secondly, based on this dataset we outline a workflow developed for scenarios realistic in plant health which utilises Bayesian modelling to estimate these parameters in the absence of a rarely available gold-standard expert dataset. In summary, our results highlight variation in the reliability of visual detection, and we provide a workflow to calculate this and facilitate optimisation of risk-based surveillance strategies in plant health.