Descriptron: Testing Artificial Intelligence for Automating Taxonomic Species Descriptions with a User-friendly Software Package

Abstract

Recent advances in Computer Vision, Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) and Large Language Models (LLMs) suggest that it may be possible to approach mimicking the ability to decode knowledge about morphology and taxonomy to describe species in an automated way. Here we test for the first time a current state-of-the-art Vision Language Model (VLM) to approximate taxonomic species descriptions in an automated manner. The test utilizes a new graphical user interface, Descriptron, that collects data about biological images and transmits this highly specialized knowledge to a VLM to decode the taxonomic knowledge encoded in labeled biological images with text. Our results indicate that current state-of-the-art VLM (GPT-4o) can produce automated species descriptions that with error correction approximate taxonomists ability to describe morphological features of species and organize them in taxonomic keys. However, the results are not without significant errors and the VLM requires the input of the taxonomists knowledge to prevent widespread hallucinations by the VLM. We find here that the taxonomist is clearly needed to both teach and closely supervise the VLM. However, the time saved by utilizing Descriptron is also significant. Taxonomists remain essential for teaching and closely supervising the VLM. The time saved by utilizing Descriptron are nevertheless already very significant. The Descriptron program and supporting example prompt files are free to use under an Apache2 License available from: https://github.com/alexrvandam/Descriptron.