Mind the Alignment Gap: A Spatial Transcriptomics Benchmark for Scientific Coding Agents

Abstract

AO_SCPLOWBSTRACTC_SCPLOWScientific coding agents are difficult to benchmark because many research tasks require executable work yet produce ambiguous or hard-to-verify outputs. Because benchmark construction requires substantial time and resources, automation offers a path to accelerating methods evaluation. We introduce an interactive framework for constructing scientificagent benchmarks from peer-reviewed papers and diagnosing agent behavior through trace inspection. We apply it as a case study in spatial transcriptomics alignment, constructing 40 tasks from SABench in which agents submit coordinate tables aligning pairs of two-dimensional tissue slices. Across 120 runs and three configurations, we compare a basic prompt, a package-aware prompt, and a full prompt with a prebuilt virtual environment. In this setting, richer package and environment context increased tool exploration but reduced the mean alignment score relative to the basic prompt (0.36 vs. 0.43; 95% CI, [-0.11,-0.03]). Trace inspection showed that added scaffolding often induced unnecessary transformations, fragile package-first workflows, and infrastructure failures. These results illustrate how specialized tooling can alter agent behavior and why scientific-agent benchmarks should evaluate agent traces and the workflows that produce them in addition to the final outputs.