henhaobio
Sign in

bioRxivBioinformaticsPreprint

Classical baselines outperform released deep-learning ITS classifiers, which collapse on ITS2 where predictions follow the flanking regions

Aaron O’BrieniD, Auguste GardetteiD, César MaríniD, Pilar ParadaiD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Background. Deep-learning classifiers for the fungal internal transcribed spacer (ITS) report accuracies above 90% and are increasingly proposed for environmental metabarcoding. That application differs from the benchmarks in two ways: surveys sequence a primer- bounded subregion, most often ITS2, rather than the full-length reference sequences the models were trained and tested on, and much of what they recover belongs to genera absent from any reference. We tested whether the reported accuracies transfer. Design. Five methods were scored on the same 5,222 queries, one sequence per genus, at two loci: the full-length ITS record, and the ITS2 subregion of that identical record. The methods are the two pretrained deep-learning classifiers distributed with MycoAI, a convolutional network and a transformer sharing a training corpus of 5.23M sequences; two reference-based methods, best-hit alignment and SINTAX; and HiTaC, a hierarchical logistic regression over k-mer counts, which we fitted ourselves to the reference the other two consult. Queries were stratified by whether the query's genus lies in the pretrained models' label space, recovered from the distributed models themselves. A novel genus is one outside that label space, and outside the reference by the same rule, so no method here can return its correct name; seen genera are the rest (2.2). Results. The design favours the pretrained models, whose queries come from the public dataset they were trained on. Even so, on full-length ITS the other three methods exceed both of them at every rank and in both strata, best-hit alignment recovering the correct family for 92.3% of seen-genus queries against 77.9% and 76.5%. Restricting the identical records to ITS2 costs the reference-based methods under four percentage points of seen-genus family accuracy and costs the two pretrained models 49.6 and 58.0, reducing them to 28.3% and 18.5%; HiTaC refitted at that locus recovers 89.2%, so neither learned classification nor the amplicon is what fails. An ablation identifies the cause. Grafting each query's unaltered ITS2 between the flanking regions of a donor record from a different phylum returns the donor's family for 34.4% of queries in the convolutional model and 63.7% in the transformer, against the query's own for 4.2% and 0.8%, from a baseline near zero where no donor is present. The predictions therefore follow the flanking regions rather than the ITS2 barcode, which accounts for the collapse and predicts the same failure for any subregion amplicon. Compounding this, on ITS2 the classifiers' confidence score all but ceases to separate novel from seen genera (AUROC 0.541 and 0.503, the latter at chance, against 0.866 for alignment identity), and the convolutional model is in addition substantially overconfident there, so for that model the failure is not detectable from its own output at all. Recommendation. Reported accuracies for such models should specify the amplicon region of the evaluation, state the length distribution of the training corpus, and be accompanied by a same-query classical baseline.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 3 (10 Sept 2026) · CC BY 4.0

MycoAI ITS classifiers trail alignment, SINTAX and HiTaC on full-length ITS and drop to 28.3% and 18.5% seen-genus family accuracy on ITS2, as predictions track flanking regions.

Published deep-learning classifiers for fungal internal transcribed spacer (ITS) barcodes quote accuracies above 90%, but on full-length reference records and test sets that overlap training, whereas environmental surveys submit ITS2 amplicons and recover many genera absent from references. This work benchmarks the released models against same-query classical baselines with novelty defined by the models' own label space, and uses a flank-swap ablation to explain the ITS2 failure.

Key results

  1. ShownEven on full-length ITS, where the design favours them, both MycoAI models (a convolutional network, CNN, and a transformer, BERT) place fewer queries in the correct family than best-hit alignment, SINTAX or HiTaC, in both the seen-genus and novel-genus strata.

    Seen/novel family recovery: alignment 92.3%/67.2%, SINTAX 92.0%/70.3%, HiTaC 92.3%/71.4% vs MycoAI-BERT 77.9%/57.7%, MycoAI-CNN 76.5%/54.3%. Every contrast between a MycoAI model and alignment, SINTAX or HiTaC is significant (exact McNemar, Holm over forty comparisons); MycoAI-BERT vs MycoAI-CNN on seen genera at full length is not (adjusted p = 0.071). Near-duplicate exclusion leaves a ten-point lead. · 3.1 Classical methods outperform the classifiers…; Abstract

  2. ShownRestricting the same records to ITS2 costs alignment, SINTAX and locus-refitted HiTaC only a few points, but both MycoAI models collapse at family rank, and MycoAI-CNN also at genus rank.

    Seen-genus family recovery fell 3.7 (alignment), 3.5 (SINTAX) and 3.1 (HiTaC) points vs 49.6 (BERT) and 58.0 (CNN), to 28.3% and 18.5%; ITS2-fitted HiTaC reached 89.2% vs 88.6% alignment and 88.5% SINTAX. Novel-genus recovery fell 6.7 points for both reference methods vs 38.9 and 40.5 for the classifiers. CNN seen-genus genus recovery went from 71.7% to 8.6%. · 3.2 On the amplicon surveys actually sequence…; 3.7 Training at the amplicon locus removes the…

  3. ShownA flank-swap ablation shows the classifiers' predictions follow the regions flanking ITS2 rather than the ITS2 barcode: with an intact query ITS2 placed between another phylum's flanks, both return the donor's family far more often than the query's own, and the transformer does so for most queries.

    Arm (iv), n = 5,221: donor's family returned for 34.4% (CNN) and 63.7% (BERT) vs the query's own for 4.2% and 0.8%; donor agreement without a donor 0.1 to 0.7%. Padding ITS2 with random sequence to 558 bp cut seen-genus family recovery from 18.5% to 6.0% (CNN) and 28.3% to 7.5% (BERT); own ITS1 alone gave 15.7% and 22.5%. · 3.3 The classifiers read the flanking regions…; 2.6 Flank ablation; Abstract

  4. ShownOn ITS2 the MycoAI family scores barely separate novel from seen genera, unlike alignment identity and the SINTAX bootstrap. MycoAI-CNN is also overconfident there, while MycoAI-BERT stays calibrated only because its scores are mostly low.

    ITS2 novelty AUROC 0.541 (BERT) and 0.503 (CNN; 95% CI 0.487 to 0.518), down from 0.688 and 0.706 on full ITS, vs 0.866 for alignment identity and 0.785 for SINTAX. A conformal flag at 5% false-novelty detects 6.1% (CNN) and 5.2% (BERT) of novel genera vs 42.7% for alignment. CNN expected calibration error 0.226 on ITS2 vs 0.065 at full length; BERT 0.019 with median score 0.171. · 3.5 On ITS2 the classifiers’ confidence loses…; G Calibration of the scores used as confidence

  5. ShownThe ITS2 deficit comes from a mismatch between training and test locus rather than from ITS2 itself: HiTaC scores high when trained and tested on the same locus and low when the loci differ, in either direction.

    HiTaC family recovery, seen/novel: fitted and tested on ITS2 89.2%/62.5%; full-ITS fit on ITS2 queries 47.4%/30.8%; ITS2 fit on full-ITS queries 67.7%/40.1%. Enlarging the full-ITS reference from 16,223 to 55,924 sequences raised full-ITS seen-genus recovery from 90.4% to 92.3% but lowered ITS2 recovery from 69.5% to 47.4%. · 3.7 Training at the amplicon locus removes the…

Methods, data and reproducibility

Five classifiers (MycoAI-CNN and MycoAI-BERT as distributed, vsearch best-hit alignment, SINTAX, and HiTaC fitted by the authors) were scored at family rank on 5,222 one-per-genus UNITE+INSD records, as full-length ITS and as the ITS2 subregion, stratified seen/novel by the label space read from the MycoAI checkpoints; a seven-arm flank ablation, novelty AUROC and calibration analyses followed.

  • Query set: one ITS record per genus from the UNITE+INSD 2024 ITS reference distributed with dnabarcoder (21.04.2024 release), as full-length ITS (median 562 bp) and ITS2 (median 169 bp) · fungi · n = 5,222 (3,330 seen-genus, 1,892 novel-genus) · Zenodo 13336328
  • Reference for alignment, SINTAX and HiTaC, mirroring the models' label space, capped at 20 sequences per genus, disjoint from queries; depth sweep references of 16,352 and 113,455 sequences · fungi · n = 56,327 sequences across 3,346 genera (full ITS); 54,108 (ITS2) · Zenodo 13336328
  • Pretrained MycoAI-CNN and MycoAI-BERT checkpoints, trained on the UNITE+INSD 2023 release (791 families, 3,695 genera) · fungi · n = about 5.23M training sequences · Zenodo 10904344
  • Flank ablation arms built from the same records (ITS2 alone, random padding, native flanks, interphylum donor flanks, single-region add-backs) · fungi · n = 5,221 records (arm iv)
  • ITSxRust re-extraction subset for the ITS2 delimitation check · fungi · n = 2,637 records
  • Genera outside the models' family space, excluded and reconciled against GBIF and Catalogue of Life · fungi · n = 252 excluded; 195 checked
Methods & tools
MycoAI-CNN, MycoAI-BERT, mycoai-its v0.0.5, vsearch v2.31.0, SINTAX, HiTaC, ITSx v1.1.3, ITSxRust v0.2.2, UNITE, GBIF, Catalogue of Life, PyTorch v2.13.0
Software & environment
Python 3.10.20 with vsearch v2.31.0, ITSx v1.1.3, ITSxRust v0.2.2, mycoai-its v0.0.5, torch v2.13.0 (unpickling default overridden), scikit-learn v1.3.2, numpy v1.26.4, biopython v1.87; pinned environment and commands in Appendix H; MIT licence.
Assay conditions & compute
Single eight-thread CPU, no GPU; inference over 5,222 queries took about 2 s (MycoAI-CNN) and 295 s (MycoAI-BERT); each HiTaC fit about ten minutes; all arms reproducible in a few CPU-hours.
Reproducibility gaps
Query sets, references and ablation arms are not deposited and must be rebuilt from UNITE with the scripts; the MycoAI package needs workarounds for a telemetry login at import and for checkpoint loading under PyTorch 2.6 or later.
Limitations (6)
  • Both models come from one group, share a training corpus and identical label spaces, and were run as distributed with default inference settings and no ITS2 fine-tuning; independently developed classifiers might behave differently. · stated by the authors
  • Every arm uses curated reference records cut to a subregion; sequencing error, primer bias and chimeras from real environmental runs are absent, so field performance is untested. · stated by the authors
  • The ablation relies on constructed chimeric and padded inputs and localizes dependence only to the flanks as a block; it does not separate ITS1 from 5.8S, identify a mechanism, or test the order-versus-composition account of why the two models differ. · stated by the authors
  • One sequence per genus discards intrageneric ITS variation and abundance weighting, so absolute accuracies are not comparable with published figures; three-fifths of seen-genus queries have a reference neighbour at 99% identity or more, so part of the classical methods' margin is retrieval. · stated by the authors
  • Near-duplicate exclusion figures appear inconsistent: Section 3.1 gives a 3.8 to 4.5 point cost for the leading methods, under one point for MycoAI and a ten-point residual lead, whereas Appendix H gives 3.8 to 6.3, 0.6 to 2.3 and about eleven. · noted in this summary
  • HiTaC, one of the top novel-genus placers at full length, gets no novelty score because its confidence filter was not trained, so whether a locus-matched fitted classifier can also flag novel genera on ITS2 is untested. · noted in this summary

Read this if you classify fungal ITS2 amplicons or evaluate pretrained barcode classifiers and want same-query baselines, label-space novelty stratification and a flank-swap ablation design.

Citations
2

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISMoon '26Heeger '19Miranda '24O’Brien '26O’Brien '26Rolling '21
74% semantic matchBioinformatics
Quinn Moon, Leho Tedersoo, Christian Griebler+3 more·bioRxiv·Sep 3, 2026