The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines

Abstract

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.

Keywords: interdisciplinary perspective; metaresearch; open science; replicability crisis; uncertainty.

Figure 1.

The multiplicity of analysis strategies arising from data preprocessing, model and method choices to obtain an estimate of the parameter of interest θ and values of the outcome variable $\mathcal{Y}$ for two research questions in epidemiology and hydroclimatology, respectively.

Figure 2.

Sources of uncertainty in explanatory, mechanistic predictive and agnostic predictive modelling. Data preprocessing, parameter, model and method uncertainty are epistemic sources of uncertainty arising from a lack of knowledge in the specification of the analysis strategy. Measurement and sampling uncertainty are random sources of uncertainty that lead to variability in the results when the same analysis strategy is applied on different datasets. The model structure describes the association between the p input variables $X_1,X_2,\dots ,X_p$ and the outcome of interest Y. θ is a parameter and e represents a probabilistic error term.

Figure 3.

The impact of random sources of uncertainty and of the multiplicity of possible analysis strategies on the replicability of research findings. The result of interest is the parameter θ in explanatory modelling, the outcome $\mathcal{Y}$ in mechanistic predictive modelling and the predictive performance in agnostic predictive modelling. The yellow colour represents the results of the chosen analysis strategy—a strategy selected because it presents the most ‘favourable’ results. It is clear that the traditional confidence interval (given by the bars around the estimate ‘x’), which only takes into account sampling uncertainty, is inadequate in capturing the true uncertainty in the estimate.

Figure 4.

Overview of solutions to the replication crisis which address the multiplicity of analysis strategies by reducing, reporting, integrating or accepting uncertainty. For an interactive version of this graphic with assorted references see https://shiny.psy.lmu.de/multiplicity/index.html.