Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Dec 14;20(1):506.
doi: 10.1186/s12909-020-02410-z.

Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews

Michael D Wolcott  1   2   3 , Nikki G Lobczowski  4   5 , Jacqueline M Zeeman  6 , Jacqueline E McLaughlin  6   4
Affiliations

Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews

Michael D Wolcott et al. BMC Med Educ. .

Abstract

Background: Situational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context.

Methods: Thirty participants-15 student pharmacists and 15 practicing pharmacists-completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes.

Results: The findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item.

Conclusions: This study provides additional SJT validity evidence by exploring participants' response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.

Keywords: Cognitive interview; Empathy; Qualitative methodology; Response process; Situational judgment test; Think-aloud protocol; Validity.

PubMed Disclaimer

Conflict of interest statement

The authors have no financial or non-financial competing interests to disclose.

Figures

Fig. 1
Fig. 1
Adaptation of the SJT response process model based on Ployhart and additional research [18, 20, 26, 28, 29]
Fig. 2
Fig. 2
A revised model of SJT response processes based on the findings from this study
See this image and copyright information in PMC

References

    1. Koczwara A, Patterson F, Zibarras L, Kerrin M, Irish B, Wilkinson M. Evaluating cognitive ability, knowledge tests, and situational judgment tests for postgraduate selection. Med Educ. 2012;46:399–408. - PubMed
    1. Patterson F, Knight A, Dowell J, Nicholson S, Cousans F, Cleland J. How effective are selection methods in medical education? A systematic review. Med Educ. 2016;50:36–60. - PubMed
    1. Campion MC, Ployhart RE, MacKenzie WI., Jr The state of research on situational judgment tests: a content analysis and directions for future research. Hum Perform. 2014;27:283–310.
    1. Chan D, Schmitt N. Situational judgment and job performance. Hum Perform. 2002;15:233–254.
    1. Lievens F, Patterson F. The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J Appl Psychol. 2011;96:927–940. - PubMed

LinkOut - more resources