3D Molecular Pretraining via Localized Geometric Generation

Abstract

Self-supervised learning on 3D molecular structures is gaining importance in data-driven scientific research and applications due to the high costs of annotating bio-chemical data. However, the strategic selection of semantic units for modeling 3D molecular structures remains underexplored, despite its crucial role in effective pre-training--a concept well-established in language processing and computer vision. We introduce Localized Geometric Generation (LEGO), a novel approach that treats tetrahedrons within 3D molecular structures as fundamental building blocks, leveraging their geometric simplicity and widespread presence across chemical functional patterns. Inspired by masked modeling, LEGO perturbs tetrahedral local structures and learns to reconstruct them in a self-supervised manner. Experimental results demonstrate LEGO consistently enhances molecular representations across biochemistry and quantum property prediction benchmarks. Additionally, the tetrahedral modeling and pretraining generalize from small molecules to larger molecular systems, validating by protein-ligand affinity prediction. Our results highlight the potential of selecting semantic units to build more expressive and interpretable neural networks for scientific AI applications.