Four Bad Habits of Modern Psychologists

Abstract

Four data sets from studies included in the Reproducibility Project were re-analyzed to demonstrate a number of flawed research practices (i.e., "bad habits") of modern psychology. Three of the four studies were successfully replicated, but re-analysis showed that in one study most of the participants responded in a manner inconsistent with the researchers' theoretical model. In the second study, the replicated effect was shown to be an experimental confound, and in the third study the replicated statistical effect was shown to be entirely trivial. The fourth study was an unsuccessful replication, yet re-analysis of the data showed that questioning the common assumptions of modern psychological measurement can lead to novel techniques of data analysis and potentially interesting findings missed by traditional methods of analysis. Considered together, these new analyses show that while it is true replication is a key feature of science, causal inference, modeling, and measurement are equally important and perhaps more fundamental to obtaining truly scientific knowledge of the natural world. It would therefore be prudent for psychologists to confront the limitations and flaws in their current analytical methods and research practices.

Keywords: NHST; Observation Oriented Modeling; inference; measurement; modeling; replication.

Figure 1

Means and standard errors for proportions of errors committed on the Stroop task for the low, medium, and high contingency conditions.

Figure 2

Proportions of errors committed on the Stroop task for the low, medium, and high contingency conditions. The results have been separated into three categories based on their ordinal patterns.

Figure 3

Example path diagrams showing a latent (ellipse) and observable (squares) variables.

Figure 4

Model representing fitness-related and generic mnemonic processing of the words “truck” and “temple.” Visual images are represented as elongated hexagons; simple predication is represented as a circle; complex judgments are represented as pentagons; memory storage is represented as hexagrams; “Ef” represents efficient cause, and “Fi” represents final cause.

Figure 5

Model representing integration of “truck” into imagined scene of survival. Visual images are represented as elongated hexagons; simple predication is represented as a circle; complex judgments are represented as pentagons; memory storage is represented as hexagrams; “if” and “else” represent standard logical operators; “Ef” represents an efficient cause; “Fo” represents a formal cause, and “Fi” represents a final cause.

Figure 6

The observed vs. predicted values for each regression model, the two regression lines, and the 95% prediction intervals around each regression line (the dotted lines).

Figure 7

The histogram of absolute value discrepancies between Model 1 and Model 2 predicted observations.