RabbitCD-HIT: Distributed High-Sensitivity Greedy Incremental Clustering for Billion-Scale Protein Sequences in Hours

Abstract

We present RabbitCD-HIT, a scalable distributed high-sensitivity greedy incremental clustering tool for ultra-large protein sequence datasets. It fully exploits cluster resources to enable efficient billion-scale clustering. RabbitCD-HIT clustered the March 10, 2026 NCBI nr snapshot (1.02 billion sequences; 487 GB FASTA) at 90\% identity within 5 hours on 64 nodes; the resulting representatives and cluster assignments are released as a public database resource. On a 256-core server, RabbitCD-HIT achieved a 2.1$\times$--6.6$\times$ speedup over CD-HIT with identical results under matched settings. Our approach addresses performance bottlenecks of traditional greedy incremental algorithms and provides an effective solution for large-scale protein sequence clustering.