Pitfalls in Post Hoc Analyses of Population Receptive Field Data

Abstract

Data binning involves grouping observations into bins and calculating bin-wise summary statistics. It can cope with overplotting and noise, making it a versatile tool for comparing many observations. However, data binning goes awry if the same observations are used for binning (selection) and contrasting (selective analysis). This creates circularity, biasing noise components and resulting in artifactual changes in the form of regression towards the mean. Importantly, these artifactual changes are a statistical necessity. Here, we use (null) simulations and empirical repeat data to expose this flaw in the scope of post hoc analyses of population receptive field data. In doing so, we reveal that the type of data analysis, data properties, and circular data cleaning are factors shaping the appearance of such artifactual changes. We furthermore highlight that circular data cleaning and circular sorting of change scores are selection practices that result in artifactual changes even without circular data binning. These pitfalls might have led to erroneous claims about changes in population receptive fields in previous work and can be mitigated by using independent data for selection purposes. Our evaluations highlight the urgency for us researchers to make the validation of analysis pipelines standard practice. Graphical abstract O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=116 SRC="FIGDIR/small/422942v3_ufig1.gif" ALT="Figure 1"> View larger version (33K): org.highwire.dtl.DTLVardef@11c0c73org.highwire.dtl.DTLVardef@7583d3org.highwire.dtl.DTLVardef@121cf6dorg.highwire.dtl.DTLVardef@b9340d_HPS_FORMAT_FIGEXP M_FIG C_FIG HighlightsO_LICircular data binning produces artifactual changes in the form of regression towards the mean C_LIO_LIAnalysis type, data properties, and circular data cleaning shape these artifactual changes C_LIO_LICircular data cleaning and sorting produce artifactual changes even without circular data binning C_LI