SegVeg: Segmenting RGB images into green and senescent vegetation by combining deep and shallow methods

Abstract

The pixels segmentation of high resolution RGB images into background, green vegetation and senescent vegetation classes is a first step often required before estimating key traits of interest including the vegetation fraction, the green area index, or to characterize the sanitary state of the crop. We developed the SegVeg model for semantic segmentation of RGB images into the three classes of interest. It is based on a U-net model that separates the vegetation from the background. It was trained over a very large and diverse dataset. The vegetation pixels are then classified using a SVM shallow machine learning technique trained over pixels extracted from grids applied to images. The performances of the SegVeg model are then compared to a three classes U-net model trained using weak supervision over RGB images with predicted pixels by SegVeg as groundtruth masks. Results show that the SegVeg model allows to segment accurately the three classes, with however some confusion mainly between the background and the senescent vegetation, particularly over the dark and bright parts of the images. The U-net model achieves similar performances, with some slight degradation observed for the green vegetation: the SVM pixel-based approach provides more precise delineation of the green and senescent patches as compared to the convolutional nature of U-net. The use of the components of several color spaces allows to better classify the vegetation pixels into green and senescent ones. Finally, the models are used to predict the fraction of the three classes over the grids pixels or the whole images. Results show that the green fraction is very well estimated (R2=0.94) by the SegVeg model, while the senescent and background fractions show slightly degraded performances (R2=0.70 and 0.73, respectively). We made SegVeg publicly available as a ready-to-use script, as well as the entire dataset, rendering segmentation accessible to a broad audience by requiring neither manual annotation nor knowledge, or at least, offering a pre-trained model to more specific use.