henhaobio
Sign in

Ecological InformaticsEcologyPeer-reviewed journal article

Large language models possess some ecologicalknowledge, but how much?

Filip DormiD, Joseph W. MillardiD, Drew W. PurvesiD, Michael HarfootiD, Oisin Mac AodhaiD

PDFDOIPreprintCodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in question answering across various domains, yet their effectiveness in ecological knowledge remains underexplored. Understanding their potential to recall and synthesize ecological information is crucial as AI tools become increasingly integrated into scientific workflows. Here, we assess the ecological knowledge of two LLMs, Gemini 1.5 Pro and GPT-4o, across a suite of ecologically focused tasks. These tasks evaluate an LLMs ability to predict species presence, generate range maps, list critically endangered species, classify threats, and estimate species traits. We introduce a new benchmark dataset to quantify LLM performance against expert-derived data. While the LLMs tested outperform naive baselines, achieving around 20 percent-age points higher accuracy in species presence prediction, they reach only a third of the mean F1 score for range map generation and improve threat classification by just around 10 points over random guessing. These results highlight both the promise and challenges of applying LLMs in ecology. Our findings suggest that domain-specific fine-tuning is necessary to improve eco-logical knowledge in LLMs. By providing a repeatable evaluation framework, our benchmark dataset will facilitate future research in this area, helping to refine AI applications for ecological science.

In-depth summary

Claude Opus 5.5 · xhigh effort · 5 Nov 2026 · from the full text of version 3 (4 Feb 2026) · CC BY 4.0

Benchmarking GPT-4o and Gemini 1.5 Pro across five closed-book ecological tasks shows solid coarse species presence and trait recall but poor fine-scale range mapping and threat classification.

Ecologists increasingly adopt off-the-shelf large language models for biodiversity synthesis, yet their parametric ecological knowledge had not been systematically benchmarked against expert databases. Evaluating closed-book performance across spatial, conservation, and trait tasks establishes where models succeed and where external retrieval or fine-tuning is required.

Key results

  1. ShownBoth LLMs outperform naive baselines on species presence/absence prediction, especially at the country level, but fail near range boundaries.

    Across 100 species (25 each of amphibians, reptiles, mammals, and birds), GPT-4o and Gemini 1.5 Pro beat baselines by ~20 percentage points and classified >98% of easy absences, but GPT-4o reached only 28.5% accuracy on boundary absences (with 85% presence accuracy vs 64.4% for Gemini 1.5 Pro) and had lower country accuracy in Africa (p < 0.001), Asia (p = 0.003), and Europe (p < 0.001). · Abstract; 3.1. Task 1: Species Presence; Appendix C.1. Difficulty of Absence Predictions; Tables

  2. ShownLLM-generated GeoJSON range maps achieve only about one-third of the spatial F1 accuracy of a deep-learning species distribution model and show low internal consistency with point predictions.

    Predicted polygons intersected expert IUCN ranges for 77/100 species (GPT-4o, 15 invalid maps) and 68/100 (Gemini 1.5 Pro, 26 invalid maps), but mean F1 scores were 20.9% (GPT-4o) and 24.7% (Gemini 1.5 Pro), roughly one-third of coordinate-only SINR, and consistency between Task 1a presences and Task 2 polygons averaged 0.274 (GPT-4o) and 0.315 (Gemini 1.5 Pro). · Abstract; 3.2. Task 2: Range Maps; Appendix E. Consistency Between Geographical…; Tables

  3. ShownBoth models miss most critically endangered bird species in high-richness countries and barely exceed random guessing when classifying IUCN threat scope and severity.

    When listing critically endangered birds by country (Task 3), responses averaged 2.3 species (max 19 vs 30 in Indonesia); on 100 IUCN threat descriptions (Task 4), scope accuracy was 48.5% (GPT-4o) and 47.5% (Gemini 1.5 Pro) vs ~33% random, and severity accuracy reached 28.3% for Gemini 1.5 Pro (71.7% cost-sensitive accuracy) vs ~16.7% random. · Abstract; 3.3. Task 3: List Threatened Species in Region; 3.4. Task 4: Threat Assessment; Tables

  4. ShownGPT-4o consistently outperforms Gemini 1.5 Pro in predicting continuous bird and mammal traits, though both models fail on mammal maximum longevity.

    Across 4 AVONET traits for 100 birds and 4 COMBINE traits for 100 mammals (Task 5), GPT-4o achieved higher R^2 and lower Normalized Mean Absolute Error on all 8 traits with 0/400 invalid responses vs 9/400 for Gemini 1.5 Pro, while both models yielded R^2 <= 0 on mammal max longevity and performed poorly on bird beak length. · 2.3.5. Task 5: Trait Identification; 3.5. Task 5: Trait Identification; Figure legends

Methods, data and reproducibility

Queried gpt-4o-2024-05-13 and gemini-1.5-pro-001 via API (temperature 0, closed-book) on 5 tasks: (1) presence/absence across 10 H3 hex centroids (~253 km^2) and countries for 100 species; (2) GeoJSON range map F1 vs IUCN and SINR; (3) country-level critically endangered bird recall; (4) IUCN threat scope/severity classification on 100 texts; and (5) estimating 4 bird (AVONET) and 4 mammal (COMBINE) traits (n=100).

  • IUCN Red List expert-derived H3 hexagonal range maps (~250-253 km^2 cells) and stratified species sample for presence/absence (Task 1) and range map (Task 2) evaluation · multiple species · n = 100 sampled species (25 per class) from 2,418 IUCN species; 10 points/species
  • IUCN Red List API records of critically endangered (CR) bird species per country (Task 3) and threat text descriptions spanning all scope-severity pairs (Task 4) · multiple species · n = up to 30 CR bird species/country; 100 unique threat severity-scope pairs
  • AVONET bird morphological trait averages (beak length, wing length, tail length, body mass) · multiple species · n = 100 randomly sampled bird species (400 queries)
  • COMBINE mammal morphological and life-history trait averages (adult body length, gestation length, max longevity, adult mass) · multiple species · n = 100 randomly sampled mammal species (400 queries)
Methods & tools
GPT-4o (gpt-4o-2024-05-13), Gemini 1.5 Pro (gemini-1.5-pro-001), SINR (Spatial Implicit Neural Representations), statsmodels, Python
Software & environment
Commercial APIs for gemini-1.5-pro-001 and gpt-4o-2024-05-13 queried August-October 2024 with temperature = 0 and max 4096 output tokens; Python with json and statsmodels (ols).
Reproducibility gaps
Exact replication of LLM queries depends on commercial API access to proprietary model snapshots (gpt-4o-2024-05-13 and gemini-1.5-pro-001).
Limitations (3)
  • The benchmark evaluates only two commercial models (GPT-4o and Gemini 1.5 Pro) in a zero-shot closed-book setup without retrieval-augmented generation, multimodal inputs, or model-specific prompt optimization. · stated by the authors
  • API inference costs restricted Tasks 1, 2, 4, and 5 to samples of 100 species or threat pairs, limiting statistical power to detect small performance differences between models. · stated by the authors
  • Reference datasets contain inherent approximation noise, including unquantified inter-assessor disagreement in IUCN range maps and threat categories and sex-averaged species-level trait values in AVONET and COMBINE. · stated by the authors

Read this if you benchmark large language models for biodiversity informatics or design retrieval-augmented workflows for conservation and macroecology.

Citations
18
FWCI
206.48
Field percentile
top 1%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISCastro '24Scheepens '24Keck '26Spillias '26Ramos '26Sarwal '23
78% semantic matchEcology
A. Castro, J. Pinto, L. Reino+2 more·Ecological Informatics·Sep 1, 2024