Stop Fooling Yourself! (Diagnosing and Treating Confirmation Bias)

Abstract

Confirmation bias (CB) is a cognitive bias that allows us to fool ourselves by selectively filtering data and distorting analyses to support favored beliefs or hypotheses. In this article, I will briefly review some classic experiments from cognitive psychology that illustrate what a powerful, pernicious, and insidious force CB is. I will then discuss how to recognize CB in our own thinking and behavior and describe specific elements of good experimental design that can mitigate its effects. These elements-such as randomization and blinding-are conceptually straightforward but often difficult in practice and therefore not as widely implemented as they should be.

Keywords: blinding; cognitive bias; confirmation bias; experimental design; masking; randomization.

Figure 1.

Which line is longer?

Figure 2.

The Ebbinghaus illusion. Which pink circle is bigger?

Figure 3.

Precision (reliability) and accuracy (validity).

Figure 4.

Stages of the scientific process at which confirmation bias can influence the outcome.

Figure 5.

Data from Table 2 (p. 67) of Rosenthal and Lawson 1964. Source code for figure: https://github.com/rickborn/eNeuroCB2024/blob/main/eNeuro_figure_code/Python/figure5_Born_eNeuro.ipynb

Figure 6.

A, Binomial distribution, illustrating a problem of balancing group sizes with simple randomization. B, Method for block randomization. Panel B adapted from Figure 2 of Kang et al., 2008. Source code for panel A: https://github.com/rickborn/eNeuroCB2024/blob/main/eNeuro_figure_code/Python/figure6A_Born_eNeuro.ipynb

Figure 7.

Binomial statistics. A, One example of a simulation of tossing a fair coin 100 times. B, Results from 2,000 such simulations. For each simulation, I recorded the number of “switches” (H followed by T or T followed by H) as well as the length of the longest run of consecutive tosses of the same value (H or T). C, The probability of runs of different lengths is described by the geometric distribution. Source code for figure: https://github.com/rickborn/eNeuroCB2024/blob/main/eNeuro_figure_code/Python/figure7_Born_eNeuro.ipynb

Figure 8.

Flow chart to guide the choice of a method for randomization of subjects. Adapted from Figure 6 of Kang et al., 2008

Figure 9.

Rigor icons for (A) randomization and (B) blinding. PNG files freely available from: https://www.ninds.nih.gov/current-research/trans-agency-activities/rigor-transparency/rigor-champions-and-opportunities/maximizing-data-transparency-rigor-icons