SegColR: Deep Learning for Automated Segmentation and Color Extraction

Abstract

Citizen science platforms like iNaturalist generate biodiversity data at an unprecedented scale, with observations on the order of hundreds of millions. However, extracting phenotypic information from these images, such as color of organisms, at such a large scale poses unique challenges for biologists. Some of the challenges are that manual extraction of phenotypic information can be subjective and time-consuming. Fortunately, with the maturation of computer vision and deep learning, there is an opportunity to automate large parts of the image processing pipeline. Here, I present SegColR, a user-friendly software package that leverages two state-of-the-art deep learning models - GroundingDINO and SegmentAnything - to enable automated segmentation and color extraction from images. The SegColR package provides an R-based interface, making it more accessible to evolutionary biologists and ecologists who may not have extensive coding experience. The SegColR pipeline allows users to load images, automatically segment them based on text prompts, and extract color information from the segmented regions. The package also includes visualization and data summarization functions to facilitate downstream analysis and interpretation of the results.