Predicting undergraduate OSCE performance using traditional and construct-driven situational judgment tests at admission

Abstract

Situational Judgement Tests (SJTs) are popular to screen for social skills during undergraduate medical admission as they have been shown to predict relevant study outcomes. Two different types of SJTs can be distinguished: Traditional SJTs, which measure general effective behavior, and construct-driven SJTs which are designed to measure specific constructs. To date, there has been no comparison of the predictive validity of these two types of SJTs in medical admission. With the present research, we examine whether the HAM-SJT, a traditional SJT, and the CD-SJT, a construct-driven SJT with an agentic and a communal scale, administered during undergraduate medical admission can predict OSCE (i.e., objective structured clinical examination) performance in a low-stakes (n_LS = 159) and a high-stakes (n_HS = 160) sample of medical students. Results showed a moderate positive relation between the communal scale of the CD-SJT and performance in OSCE stations with trained patients in the high-stakes sample (r =.20, p =.009). This SJT had also an incremental value in predicting the OSCE performance above and beyond GPA (i.e., grade point average), a science test (i.e., HAM-Nat), and gender (ß = 0.18, 95% CI [0.03; 0.33], p =.020). That is, individuals who chose more communal behavioral responses in the SJT were rated more favorably in interactions with trained patients in the OSCE. A comparable correlation coefficient was observed for the HAM-SJT when controlling for range restriction due to admission (r_raw = 0.14 vs. r_controlled = 0.20). Our research provides a first indication for the predictive validity of construct-driven SJTs in high-stakes undergraduate medical admission.

Keywords: Admission; Agency; Communion; OSCE; Situational judgement tests.

Fig. 1

Flowchart of detection for low-stakes (top) and high-stakes sample

Fig. 2

Example items for all SJTs with construct level for CD-SJT in parentheses