Reticulation-Aware Dissimilarities for Phylogenetic Networks

Abstract

Dissimilarity measures play a critical role in phylogenetics. They help quantify the accuracy of inferences when the ground truth is known, allow comparisons of inferences obtained using different methods, and provide a tool for summarizing sets of phylogenies. A wide array of dissimilarity measures has been introduced for phylogenetic trees over the past half-century, and they vary in their mathematical properties, empirical behavior, and computational complexity. When the evolutionary history involves reticulate processes, such as hybridization and horizontal gene transfer, it is represented by phylogenetic networks. Although the study of phylogenetic network dissimilarity is less established than its tree counterpart, the past two decades have seen an increasing number of measures developed for phylogenetic networks. However, while some have established mathematical properties---for example, metricity on certain classes of phylogenetic networks---none properly separates reticulation disagreement from tree-like disagreement. This is the rationale behind the introduction of the hybrid ancestry error (HAE) measure by Dai and Molloy. The HAE measure, however, is too simplistic to distinguish different reticulation scenarios. In this work, we propose a novel family of network dissimilarity measures that explicitly account for reticulation differences while being more informative than HAE. We prove that our measures are pseudometrics on rooted binary phylogenetic networks on a fixed taxon set, with two instantiations yielding metrics on binary level-1 networks. Empirically, the measures remain sensitive to reticulation errors as network size grows and respond to changes in hybrid identity and parental placement. Calibration experiments further characterize the measures' behavior under network edits of various magnitudes. Data Availability: The dissimilarities are implemented in the PhyNetPy library, available at https://github.com/NakhlehLab/PhyNetPy. All the data simulated for the study is available at https://doi.org/10.5281/zenodo.23933421. Funding: This work was supported in part by the NSF grants DMS/NIGMS-2153704 and DBI-2420499. Declaration of AI-assisted technologies: During the preparation of this work, the authors used OpenAI's Codex (GPT-6 Astra) and Anthropic's Claude (Fable 5) to assist with editing manuscript text, developing proofs and analysis code, and preparing figures. The authors reviewed and edited the resulting material, verified the analyses, and take full responsibility for the content of this article.