Published estimates of group differences in multisensory integration are inflated

Abstract

A common measure of multisensory integration is the McGurk effect, an illusion in which incongruent auditory and visual speech are integrated to produce an entirely different percept. Published studies report that participants who differ in age, gender, culture, native language, or traits related to neurological or psychiatric disorders also differ in their susceptibility to the McGurk effect. These group-level differences are used as evidence for fundamental alterations in sensory processing between populations. However, there is high variability in the McGurk effect within groups: some participants never report the illusion while others always do. High within-group variability means that sample sizes much larger than those in the published literature are necessary to accurately estimate between-group differences. Using empirical data and simulations, we show that a typical study with a sample size of 15 participants per group and a true group difference of 10% will report a group difference of 31%, an effect inflation of three-fold. Reducing the effect inflation by half requires increasing the sample size ten-fold, to 150 participants per group. The combination of within-group variability and small sizes explains two prominent features of the McGurk literature. First, group differences are inflated and highly variable across studies. Second, follow-up studies fail to replicate published reports of large between-group susceptibility differences because initial effect-size estimates are driven by statistical inflation rather than true group differences. These replication failures are especially problematic in studies of clinical populations because they have real-world consequences. Due to the high within-group variability, caution is essential when using the McGurk effect to assess deficits in clinical groups or propose treatment plans, such as training to improve sensory processing.