henhaobio
Sign in

PaleobiologyPaleontologyPeer-reviewed journal article

Addressing the open world: detecting and segmenting pollen on palynological slides with deep learning

Jennifer T. FengiD, Sandeep Puthanveetil SatheesaniD, Shu KongiD, Timme H. DondersiD, Surangi W. PunyasenaiD

PDFDOIPreprintCode
In-depth full-text digest available · Jump to digest ↓

Abstract

In the "open world", categorical classes are imbalanced, test classes are not known a priori, and test data are captured across different domains. Paleontological data can be described as open-world, as specimens may include new, unknown taxa, and the data collected, such as measurements or images, may not be standardized across different studies. Fossil pollen analysis is one example of an open-world problem in paleontology. Pollen samples capture large numbers of specimens, including not only common types but also rare and even novel taxa. Pollen is diverse morphologically and features can be altered during fossilization. Additionally, there is little standardization in the methods used to capture and catalog pollen images and most collections are mounted on microscope slides. Therefore, generalized workflows for automated pollen analysis require techniques that are robust to these differences and can work with microscope images. We focus on a critical first step, the detection of pollen specimens on a palynological slide and review how existing methods can be employed to build robust and generalizable analysis pipelines. First, we demonstrate how a mixture-of-experts approach - the fusion of a general pollen detector with an expert model trained on minority classes - can be used to address taxonomic biases in detections, particularly the missed detections of rarer pollen types. Second, we demonstrate the efficiency of domain fine-tuning in addressing domain gaps - differences in image magnification and resolution across microscopes, and of taxa across different sample sources. Third, we demonstrate the importance of continual learning workflows, which integrate expert feedback, in training detection models from incomplete data. Finally, we demonstrate how cutting-edge segmentation models can be used to refine and clean detections for downstream deep learning classification models. Non-technical AbstractFossil pollen analysis presents challenges that result from the complexity of the "open world". It is not possible to anticipate all the taxa that will be encountered in geologic samples. Each sample has the potential to produce new species and introduce specimens with unique shapes and sizes. Pollen embodies diverse morphologies, its appearance can be altered depending on the conditions under which it was fossilized, and there is little standardization in the microscopy equipment used to image and catalog pollen images. AI applications in palynology need to address this complexity. We applied four machine learning methods (mixture-of-expert models, domain fine-tuning, continual learning, and foundation models for segmentation) to fossil pollen data to demonstrate their effectiveness in the detection and isolation of pollen specimens on a pollen sample slide.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (5 Jan 2025) · CC BY 4.0

Fusing a small-grain expert detector, fine-tuning across microscopes, human-in-the-loop continual learning and zero-shot SAM-2 segmentation improve detection and isolation of fossil pollen on scanned slides.

Automated fossil pollen counting needs detectors that cope with long-tailed and partly unknown taxa and with images that differ across microscopes, yet prior detectors were reported at over 90% accuracy on just 10 to 11 pollen types, and earlier slide pipelines merged focal planes into a single 2D image. Here the detector is trained on unfused multi-plane stacks with annotations covering 129 taxonomic types, and four open-world strategies are tested against rare small grains, domain shift and incomplete labels.

Key results

  1. ShownFine-tuning the PAL IV-trained detector on a small annotated sample from a second microscope (a 400x slide scanner) substantially improved detection there and outperformed a model trained from scratch on the same images.

    Without fine-tuning, the detector on PAL 1999 scanner images gave mean average precision (mAP) 32.56%, maximum recall 82% and 10% precision at 80% recall. After fine-tuning: mAP 65.99%, 46% precision at 80% recall, maximum recall 93%. Training from scratch on PAL 1999 alone: mAP 59.27%. New-domain annotation covered about 5% of each slide's area. · Transfer Learning Across Imaging Domains; Transfer Learning Across Imaging Domains; Discussion; Imaging

  2. ShownFusing a small-grain expert detector (SGDM, fine-tuned on three small-grained taxa) with the general detector (GPDM) recovered small, transparent grains the general model missed, with a modest overall gain.

    On PAL IV validation data, mAP rose from 73.21% to 75.09% and maximum recall from 93% to 95%. Recall at 20% precision: Urticaceae-Moraceae 69% to 81%, Melastomataceae 75% to 100%, Cecropia 0% to 50%. Acalypha, Vallea and Weinmannia, withheld from SGDM training, also improved (values shown only graphically). · Mixture-of-Experts Technique; Mixture-of-Experts; Figure legends

  3. SuggestedIterative fine-tuning across three slide-level rounds (ground-truth labels in TP0, then detector outputs with expert verification simulated from ground-truth labels in TP1 and TP2) raised mAP in each round on the slide-scanner data.

    Within-round mAP rose from 34.11% to 58.72% (TP0), 48.25% to 62.60% (TP1) and 61.57% to 76.01% (TP2). On a shared validation set pooling all three rounds, mAP went from 41.10% (TP0 model) to 59.93% (TP2 model). Detections were kept if intersection over union (IoU) with ground truth exceeded 0.3; missed grains were not added. · Continual Learning with Human-in-the-Loop; Continual Learning with Human-in-the-Loop

  4. ShownThe general detector, trained directly on unfused focal-plane stacks, reached high recall but under-detected small, transparent grains; rarity alone did not explain the misses.

    PAL IV: maximum recall 93%, mAP 73.21%; the authors report about 90% recall at 20% precision across their experiments. Cecropia (19 training examples): none of 4 validation images detected at 20% precision, while Polylepis spp. (also 19) had 100% recall. Ericaceae, Rumex and Lupine, with ten or fewer training examples, were detected at 100% recall. · General Pollen Detection Model (GPDM); Discussion

  5. SuggestedSAM-2, never trained on pollen, produced masks covering nearly whole grains from a single centre-point prompt taken from the detector output.

    Qualitative only (Fig. 8): the highest-scoring SAM-2 masks followed grain outlines; adding four negative corner prompts did not improve results. Masks were dilated (5 x 5 kernel, 1 iteration) to include the wall and ornamentation. No segmentation accuracy metric is reported. · Zero-shot Segmentation; Zero-shot Segmentation

Methods, data and reproducibility

A ResNet34-backbone convolutional network predicting a per-pixel pollen probability map and a distance-transform map (whose peaks mark grain centres) was trained on unfused multi-focal-plane image stacks. Four add-ons were tested: fusion with a small-grain expert via non-maximum suppression, fine-tuning to a second microscope, simulated human-in-the-loop continual learning and zero-shot SAM-2 segmentation.

  • Laguna Pallcacocha (Ecuadorian Andes) core PAL IV, 20th century; slides imaged on a Leica DM 6000 B at 630x (0.146 μm/pixel), 7 or 9 focal planes; used for GPDM and SGDM · plants (fossil pollen and fern/lycopod spores) · n = 19 slides; 2986 image stacks; 3191 annotated specimens
  • Laguna Pallcacocha core PAL 1999, Holocene; Hamamatsu NanoZoomer 2.0 HT at 400x (0.225 μm/pixel), 9 focal planes; used for domain transfer and continual learning · plants (fossil pollen and fern/lycopod spores) · n = 3 slides; 300 image stacks per slide (900 total); 794 annotated specimens
  • Small-grain expert subset of PAL IV: Urticaceae-Moraceae, Cecropia, Melastomataceae (train/validation); Acalypha, Vallea, Weinmannia held out as novel test taxa · plants (fossil pollen) · n = 132/32, 19/4 and 25/4 specimens (train/validation)
Methods & tools
ResNet34, Segment Anything Model 2 (SAM-2), Bio-Formats (Python), Labelme, MATLAB, Hamamatsu NanoZoomer 2.0 HT, NDP.view2, Leica DM 6000 B, LAS 4.12 PowerMosaic, Docker, Apptainer, Jupyter Notebooks
Software & environment
Python CLIs containerized with Docker (Apptainer on HPC), Apache 2.0 licence; tested on macOS (Apple M and Intel chips), NCSA Delta HPC and the Radiant cloud. No package versions given.
Reproducibility gaps
Image stacks, annotations and trained models are said to be archived at the Illinois Data Bank (cited as Feng et al. 2025), but no URL or DOI is printed; training hardware, runtimes and software versions are not stated.
Limitations (5)
  • Rare taxa can still be missed by chance depending on which images fall in training versus validation; Loranthaceae and Thalictrum were missed entirely by both the general and small-grain detectors. · stated by the authors
  • The confidence threshold trades missed against false detections and must be chosen per application; lowering it raises recall but adds false detections. · stated by the authors
  • A trained detector is likely still needed upstream of SAM-2 to separate pollen from the varied organic debris on palynological slides. · stated by the authors
  • All data come from cores of one Andean lake with the same taxonomic composition, so transfer was tested mainly across imaging setups, not across new floras or sites; the human-in-the-loop step was simulated with ground-truth labels. · noted in this summary
  • Per-taxon small-grain gains rest on few validation grains (4 Cecropia, 4 Melastomataceae) from one split without repeats; SAM-2 masks were judged only visually; the PAL 1999 training size is given as both 900 and 800 images. · noted in this summary

Read this if you build object detectors for microfossils or other slide-mounted specimens and want tested recipes for rare small classes, microscope-to-microscope transfer and iterative annotation.

Citations
8
FWCI
2.83
Field percentile
top 8%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISSevillano '20Adaïme '23Rodgers '25Adaïmé '26Yu '23Carvalho '19
68% semantic matchPlant Biology
V. Sevillano, K. Holt, J. L. Aznarte·bioRxiv·Feb 14, 2020