Advancing Knotted Protein Design with ESM3: Guided Generation and Topological Insights

Abstract

Multimodal protein language models have transformed protein design, yet their capacity to capture complex topological features remains poorly understood. We use knotted proteins, rare structures in which the backbone forms a nontrivial topological knot, as a test case to probe this capacity using ESM3, a generative protein language model. Topology-aware guided decoding strongly enriches ESM3 outputs for knotted topologies, producing structures classified as knotted at an 89% success rate (95% CI: 81- 94%), compared to ~0.5% for unguided diffusion-based approaches. A confidence analysis shows that freshly generated artificial knots have lower ESM3 pLDDT and pTM than real knotted proteins evaluated under the same pipeline, motivating a cautious interpretation of generated examples as model samples pending independent validation. In contrast, the robustness analyses on real knotted proteins are high-confidence: on average 84% of the protein sequence must be altered before the knot breaks, and the loss follows a sharp threshold rather than gradual degradation. Strikingly, structural drift accumulates well before topological disruption, suggesting that topology is more robust than specific three-dimensional arrangement. These findings position knotted proteins as a useful probe of how generative protein models represent rare, global structural features.