Autonomous generalizability auditing of scientific foundation models

Determining where scientific foundation models generalize beyond their development data is essential for reliable use, but aggregate benchmarks obscure performance limits and rigorous audits are difficult to conduct routinely. Here, we introduce AutoSPECTRA, a coding-agent protocol for systematic generalizability auditing. To inform its design, we reviewed 38 studies across five biological domains and evaluated coding agents, which generated hypotheses and empirical analyses but applied inconsistent standards for judging evidence. We then structured AutoSPECTRA around measurable generalization axes, prespecified analyses, controlled comparisons and independent confirmation. Across four biological foundation models, audits linked performance to protein sequence composition and relatedness to pretraining data (ESMFold2), representation of perturbations in training (STATE), effective spatial resolution (CONCH), and residue-level genomic, evolutionary and structural context (AlphaMissense). Agents also evaluated alternative explanations and rejected unsupported hypotheses. AutoSPECTRA provides a systematic approach to discovering and testing the limits of scientific foundation models.