Bivariate multilevel meta-analysis of log response ratio and standardized mean difference for robust and reproducible environmental and biological sciences

Abstract

Meta-analytic modelling plays a pivotal role in synthesizing research and informing relevant policies. Yet researchers face many analytical challenges. In environmental and biological sciences, one of the most common yet unrecognised issues is the selection between two common effect size metrics, log response ratio (lnRR) and standardized mean difference (SMD); these two are the most popular and alternative effect sizes. Having to choose between them creates room for analytical flexibility, which is susceptible to researcher degrees of freedom. Another common issue is failure to deal with statistical dependence between effect sizes, resulting in invalid inferences on evidence. We propose addressing these two issues through the joint synthesis (dual use) of lnRR and SMD. Using 75 meta-analyses, including 3,887 environmental/biological primary studies ([~]20,000 effect sizes), we show a high false positive rate (40%) in conventional meta-analytic practices (random-effects model) compared to the proposed bivariate multilevel meta-analysis of lnRR and SMD along with robust variance estimation. Relying solely on either lnRR or SMD results in non-trivial discrepancies in detecting statistically significant effects (18%) and occasional inconsistencies in sign (9%). Discrepancies in interpreting effect size, heterogeneity, and publication bias are prevalent between models using lnRR and SMD (e.g., 52% for publication bias). In contrast, bivariate synthesis of lnRR and SMD yields substantial information gain, reducing standard error in effect size estimates by 29%, equivalent to adding 40 additional effect sizes. We present a user-friendly website with a step-by-step implementation guide. Our proposed robust approach aspires to improve meta-analytic modelling using lnRR and SMD in environmental and biological evidence synthesis, amplifying their reproducibility and credibility.