Probing isn't Causality in Molecular Models

Abstract

Linear probes establish whether a chemical variable can be recovered from a molecular representation; they do not establish how that variable contributes to subsequent predictions. We replace a concept-associated scalar component at an intermediate block and measure the change through a separately fitted downstream outcome readout. Comparisons with random directions and architecture-matched random encoders distinguish recoverability, intervention selectivity, and a pretraining-associated selectivity advantage. In Uni-Mol, block-12 total-distortion edits predict the observed energy-change sign for 97.6\% of held-out pairs, with Spearman $\rho=0.787$. Structure-clustered intervals for the primary direction and encoder contrasts exclude zero. A stricter 20-direction sensitivity analysis does not survive correction across six cells. A post-review analysis finds substantial concept--outcome alignment; intervention effects persist after removal of the same-block outcome direction, but this does not establish that subsequent encoder computation is necessary. In MolCLR, pretraining improves dipole recovery by $0.080\,R^2$, whereas concept-minus-random sign-accuracy contrasts are $0.029$ and $-0.015$ at the two tested blocks. Their paired uncertainty is unresolved, so they do not establish an absence of selective effects. We propose an operational evidence ladder whose conclusions are conditional on the concept, intervention, control family, and fitted readout.