PolyFold: Homology Filtering Approach Alters Apparent Structure Prediction Model Performances

As structure prediction models become increasingly central to biological research and drug discovery, rigorous and reproducible benchmarking is critical to assess model performance and generalization. PolyFold is an end-to-end workflow automation and benchmarking framework that streamlines input file creation, batch automation, and a more conservative evaluation of model performances. We conduct a head-to-head comparison of Boltz-2 and OpenFold 3 - leading open-source alternatives to AlphaFold 3. We first curate an evaluation set of several thousand high-quality PDB structures, homology-filtered against both models' training sets, then automate inference and evaluation using PolyFold. Using a cluster analysis, we directly connect features of the underlying structures to model performance profiles, demonstrating the nonrandom nature of failure and success modes. While initial analyses using conventional homology filtering techniques show performance differences favouring Boltz-2, we show that using minimum chain fractional identity for novelty filtering can cause residual training-data leakage in multichain complexes, inflating apparent performances. To address this, we introduce a length-weighted average fractional identity (WAFI) for whole-complex homology filtering. This provides a more conservative evaluation by reducing residual leakage. While Boltz-2 maintains higher aggregate metric averages for categories such as nucleic acid-containing structures, membrane-bound proteins, and antibody-antigen complexes, the performance gap significantly narrows or disappears for effectively all partitions after implementing WAFI filtering, demonstrating the need for more stringent conventions in homology filtering. Both models still struggle with homomultimers, small molecule realism, and structures with non-canonical amino acids, demonstrating persisting limitations in the field. In summary, PolyFold enables practitioners to rapidly compare models on a variety of inference tasks and supports the broader adoption of accessible computational tools to facilitate biomolecular science.