Are our 95% CIs only worth 45% confidence? A heterogeneity-aware method for extending inference

O_LIWhen multiple studies on the same research question, or multiple analyses of the same dataset are summarized in a meta-analysis, our confidence intervals (CIs) are put to a relentless test of reliability. While we might hope that 95% of the CIs would contain the meta-analytic mean value (and therefore presumably also the true value), a recent meta-analysis of 512 meta-analyses in ecology and evolution suggests that only a sobering 45% of them do. C_LIO_LIAs this paradox of overconfidence continues to confuse researchers, we attempt to explain where most of the heterogeneity in findings might be coming from. Much of it is unsurprising, because conventional 95% CIs reflect only sampling noise and ignore other sources of error such as arbitrary analysis decisions ("model uncertainty"). Being aware of multiple sources of error beyond sampling noise, the replication crisis logically follows from an anticonservative statistical practice that allows for overinterpretation beyond the legitimate inference space, which in fact is narrower than commonly acknowledged. C_LIO_LIWe explain how to calculate extended confidence intervals (CIext) that also cover other sources of biological and analytical heterogeneity, and we clarify which CIext is valid for which extended inference space. We further show how multiple versions of analysis of the same dataset can be merged into a many-analyses meta-analysis (MAMA) which yields a CI that accounts for two sources of error. C_LIO_LIAt the same time, we recognize that high heterogeneity estimates can be artefactual, arising from "comparing apples and oranges" (either in terms of statistical metric or biological interpretation). The estimate of only 45% deserved confidence therefore appears far too pessimistic. Overall, we argue for greater caution in interpreting both confidence intervals and heterogeneity estimates, and for clearer separation of comparable from non-comparable evidence. C_LI