Michigan ZoomIN: validating crowd-sourcing to identify mammals from camera surveys

Abstract

Camera trap studies have become a popular medium to assess many ecological phenomena including population dynamics, patterns of biodiversity, and monitoring of endangered species. In conjunction with the benefit to scientists, camera traps present an unprecedented opportunity to involve the public in scientific research via image classifications. However, this engagement strategy comes with a myriad of complications. Volunteers vary in their familiarity with wildlife, and thus, the accuracy of user-derived classifications may be biased by the commonness or popularity of species and user-experience. From an extensive multi-site camera trap study across Michigan U.S.A, images were compiled and identified through a public science platform called Michigan ZoomIN. We aggregated responses from 15 independent users per image using multiple consensus methods to assess accuracy by comparing to species identification completed by wildlife experts. We also evaluated how different factors including consensus algorithms, study area, wildlife species, user support, and camera type influenced the accuracy of user-derived classifications. Overall accuracy of user-derived classification was 97%; although, several canid (e.g., Canis lupus, Vulpes vulpes) and mustelid (e.g., Neovison vison) species were repeatedly difficult to identify by users and had lower accuracy. When validating user-derived classification, we found that study area, consensus method, and user support best explained accuracy. To continue to overcome stigma associated with data from untrained participants, we demonstrated their value by showing the accuracy from volunteers was comparable to experts when classifying North American mammals. Our hierarchical workflow that integrated multiple consensus methods lead to more image classifications without extensive training and even when the expertise of the volunteer was unknown. Ultimately, adopting such an approach can harness broader participation, expedite future camera trap data synthesis, and improve allocation of resources by scholars to enhance performance of public participants and increase accuracy of user-derived data.