Conclusions Drawn From Neural Network to Brain Alignment Depend Strongly on the Chosen Similarity Measure

Deep neural networks are widely used to model biological perception and behavior, making the similarities and differences between artificial and biological systems consequential. If a principle (e.g. self-supervised learning) produces a model resembling biology, this is taken as evidence the same principle shapes the biological system. But what it means for an artificial system to be similar to a biological one is complex. A popular approach compares representations of identical stimuli using a similarity measure like Representational Similarity Analysis. Yet scientific questions rarely specify which measure is appropriate, raising a key question: do conclusions depend on this choice? Focusing on vision, we show that measure choice influences both hierarchical correspondence between systems and the ranking of which artificial systems are most biological. Reanalyzing prior studies, we find the choice hugely consequential: models best under one measure are often worst under another, and prior conclusions can flip or dissolve. Different metrics capture fundamentally different aspects of similarity, warranting healthy skepticism toward such comparisons.