Selective attention in hypothesis-driven data analysis

Abstract

When analyzing the results of an experiment, the mental focus on a specific hypothesis might prevent the exploration of other aspects of the data, effectively blinding one to new ideas. To test this notion, we performed an experiment in which we asked undergraduate students to analyze a fictitious dataset. In addition to being asked what they could conclude from the dataset, half of the students were asked to also test specific hypotheses. In line with our notion, students in the hypothesis-free group were almost 5 times more likely to observe an image of a gorilla when simply plotting the data, a proxy for an initial step towards data analysis. If these findings are representative also of scientific research as a whole, they warrant concern about the current emphasis on hypothesis-driven research, especially in the context of information-rich datasets such as those now routinely created in the biological sciences. Our work provides evidence for a link between the psychological effect of selective attention and hypothesis-driven data analysis, and suggests a hidden cost to having a hypothesis when analyzing a dataset.