Development of the Paranormal and Supernatural Beliefs Scale using classical and modern test theory

Abstract

Background: This study describes the construction and validation of a new scale for measuring belief in paranormal phenomena. The work aims to address psychometric and conceptual shortcomings associated with existing measures of paranormal belief. The study also compares the use of classic test theory and modern test theory as methods for scale development.

Method: We combined novel items and amended items taken from existing scales, to produce an initial corpus of 29 items. Two hundred and thirty-one adult participants rated their level of agreement with each item using a seven-point Likert scale.

Results: Classical test theory methods (including exploratory factor analysis and principal components analysis) reduced the scale to 14 items and one overarching factor: Supernatural Beliefs. The factor demonstrated high internal reliability, with an excellent test-retest reliability for the total scale. Modern test theory methods (Rasch analysis using a rating scale model) reduced the scale to 13 items with a four-point response format. The Rasch scale was found to be most effective at differentiating between individuals with moderate-high levels of paranormal beliefs, and differential item functioning analysis indicated that the Rasch scale represents a valid measure of belief in paranormal phenomena.

Conclusions: The scale developed using modern test theory is identified as the final scale as this model allowed for in-depth analyses and refinement of the scale that was not possible using classical test theory. Results support the psychometric reliability of this new scale for assessing belief in paranormal phenomena, particularly when differentiating between individuals with higher levels of belief.

Keywords: Anomalous beliefs; Classical test theory; Factor analysis; Modern test theory; Paranormal beliefs; Rasch analysis; Rating scale model; Scale; Scale development; Supernatural.

Fig. 1

PCA score plot of all responses to the paranormal scale as a function of respondent group. Figure plots participants’ responses to the scale items against the two principal components that represent the largest variability among the two groups, to provide a visual indication of separation (or lack thereof) between the groups

Fig. 2

Raincloud plot of mean scale scores given as a function of respondent group. Figure presents mean Likert scores (from 1 to 7) for all items on the scale, with individual mean scores per participant shown for each group, and a histogram showing the distribution of mean scale scores for each group

Fig. 3

Test–retest reliability analysis as a function of respondent group. Pearson’s correlations between participants’ individual total scores at time one and time two shown for each group

Fig. 4

Item characteristic curve for item 1 using the 4-point scoring method. Curves represent the probability of selecting a category along the latent trait. Category 0 = “strongly disagree”, Category 1 = “disagree”, Category 2 = “agree”, Category 3 = “strongly agree”

Fig. 5

Item characteristic curve for item 3 in the reduced scale using the 4-point scoring method. Curves represent the probability of selecting a category along the latent trait. Category 0 = “strongly disagree”, Category 1 = “disagree”, Category 2 = “agree”, Category 3 = “strongly agree”

Fig. 6

Person-item map for the 13-item scale. Figure displays the location of person traits and item difficulties along the same latent dimension (paranormal belief). The person traits are located on the scale from left (low belief) to right (high belief). Locations of item difficulties are presented as solid circles, and thresholds of adjacent category locations are presented as open circles. The item parameters are located on the scale from least difficult (left) to most difficult (right)

Fig. 7

Single-node Rasch tree. Note: Figure shows differential item functioning analysis conducted on the covariates of age, gender (male or female), ethnicity (white background or BME background), education, and discipline. No differential item functioning was identified. Item number is represented on the x axis in both plots. Item difficulty is represented on the y axis of the top plot (higher values represent higher item difficulty), and item threshold parameters are shown on the y axis of the lower plot (with the lightest shade representing the ‘strongly agree’ response category)

Fig. 8

Test–retest reliability analysis for the Rasch scale as a function of respondent group. Pearson’s correlations between participants’ individual total scores at time one and time two shown for each group

Fig. 9

Correlations between respondents’ individual total scores. Pearson’s correlations between respondents’ total scores on the classical test theory and modern test theory scales, as a function of respondent group