Evaluation of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests to Identify Successful Residents

Abstract

Importance: The ability to identify candidates who will thrive and successfully complete their residency is especially critical for general surgery programs.

Objective: To assess the extent to which 3 screening tools used extensively in industrial selection settings-emotional intelligence (EQ), personality profiles, and situational judgment tests (SJTs)-could identify successful surgery residents.

Design, setting, and participants: In this analysis, personality profiles, EQ assessments, and SJTs were administered from July through August 2015 to 51 postgraduate year 1 through 5 general surgery residents in a large general surgery residency program. Associations between these variables and residency performance were investigated through correlation and hierarchical regression analyses.

Interventions: Completion of EQ, personality profiles, and SJT assessments.

Main outcomes and measures: Performance in residency as measured by a comprehensive performance metric. A score of zero represented a resident whose performance was consistent with that of their respective cohort's performance; below zero, worse performance; and greater than zero, better performance.

Results: Of the 61 eligible residents, 51 (84%) chose to participate and 22 (43%) were women. US Medical Licensing Examination Step 1 (USMLE1), but not USMLE2, emerged as a significant factor (t2,49 = 1.98; β = 0.30; P = .03) associated with overall performance. Neither EQ facets nor overall EQ offered significant incremental validity over USMLE1 scores. Inclusion of the personality factors did not significantly alter the test statistic and did not explain any additional portion of the variance. By contrast, inclusion of SJT scores accounted for 15% more of the variance than USMLE1 scores alone, resulting in a total of 25% of the variance explained by both USMLE1 and SJT scores (F2,57 = 7.47; P = .002). Both USMLE1 (t = 2.21; P = .03) and SJT scores (t = 2.97; P = .005) were significantly associated with overall resident performance.

Conclusions and relevance: This study found little support for the use of EQ assessment and only weak support for some distinct personality factors (ie, agreeableness, extraversion, and independence) in surgery resident selection. Performance on the SJT was associated with overall resident performance more than traditional cognitive measures (ie, USMLE scores). These data support further exploration of these 2 screening assessments on a larger scale across specialties and institutions.