Three Strategies for the Critical Use of Statistical Methods in Psychological Research

Abstract

We present three strategies to replace the null hypothesis statistical significance testing approach in psychological research: (1) visual representation of cognitive processes and predictions, (2) visual representation of data distributions and choice of the appropriate distribution for analysis, and (3) model comparison. The three strategies have been proposed earlier, so we do not claim originality. Here we propose to combine the three strategies and use them not only as analytical and reporting tools but also to guide the design of research. The first strategy involves a visual representation of the cognitive processes involved in solving the task at hand in the form of a theory or model together with a representation of a pattern of predictions for each condition. The second approach is the GAMLSS approach, which consists of providing a visual representation of distributions to fit the data, and choosing the best distribution that fits the raw data for further analyses. The third strategy is the model comparison approach, which compares the model of the researcher with alternative models. We present a worked example in the field of reasoning, in which we follow the three strategies.

Keywords: GAMLSS; data visualization; model comparison approach; multilevel analysis; null hypothesis.

Figure 1.

Distribution of reaction times in the four conditions, represented with boxplots for each condition and with the data of each individual in each condition (notches around the median represent approximate 95% confidence intervals).

Figure 2.

Visual representation of the cognitive processes involved in each condition and predictions for the outcome variable in each condition.

Figure 3.

Probability density functions of the selected conditional responses. “A” shows the (nonparametric) kernel density estimate of distribution of responses (“fixation time” in ms) to the two-level factor “law” (“negative conjunction” and “negative disjunction”) in the two-level factor “context” (“abstract” and “with cue”). “B” shows the distribution of responses (“fixation time” in ms) given to each set of four items (Items 1-4 and 4-8) used in each “law” factor. The first row in “B” shows the density plots shown in the left panel of “A” as separate plots. The second row in “B” shows the density plots shown in the right panel of “A” (“Y” corresponds to “fixation time”). In addition to the kernel densities, the (parametric) density function (solid) lines of the distributions fitted to the conditional responses are also shown. In all cases, the actual observations are shown as “rugs” in the X axis. “C” shows the RTs associated to each of the four experimental items in each “law” condition.