Beyond performance: How design choices shape chemical language models

Chemical language models (CLMs) have shown strong performance in molecular property prediction and generation tasks. However, the impact of design choices, such as molecular representation format, tokenization strategy, and model architecture, on both performance and chemical interpretability remains underexplored. In this study, we systematically evaluate how these factors influence CLM performance and chemical understanding. We evaluated models through finetuning on downstream tasks and probing the structure of their latent spaces using simple classifiers and dimensionality reduction techniques. Despite similar performance on downstream tasks across model configurations, we observed substantial differences in the structure and interpretability of their internal representations. SMILES molecular representation format with atomwise tokenization strategy consistently produced more chemically meaningful embeddings, while models based on BART and RoBERTa architectures yielded comparably interpretable representations. These findings highlight that design choices meaningfully shape how chemical information is represented, even when external metrics appear unchanged. This insight can inform future model development, encouraging more chemically grounded and interpretable CLMs. Scientific ContributionThis study systematically evaluates how core design choices influence chemical language models. Although the performances on downstream tasks were often similar across configurations, we observed substantial differences in internal representations with atomwise tokenized SMILES representations producing more chemically structured latent spaces than representations based on SELFIES. By clarifying the effects of molecular representation format and tokenization strategy, our findings provide actionable guidance for the more informed and interpretable design of future CLMs. Graphical Abstract O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=80 SRC="FIGDIR/small/655735v1_ufig1.gif" ALT="Figure 1"> View larger version (23K): org.highwire.dtl.DTLVardef@11c387org.highwire.dtl.DTLVardef@3be797org.highwire.dtl.DTLVardef@e197c2org.highwire.dtl.DTLVardef@b264e2_HPS_FORMAT_FIGEXP M_FIG C_FIG