Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training

Abstract

Aim: The study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination.

Methods: In this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05.

Results: Mean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1.

Conclusions: The change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant's future performance in residency.

Keywords: CREOG; Step 1; USMLE; in-service examination; resident selection.

Figure 1.

Scoring system. Scoring used for Medical Student Performance Evaluation (MSPE)/Dean’s letter and clerkship grade.

Figure 2.

Summary of dataset (n = 75).

Figure 3.

Correlation of USMLE and CREOG scores. CREOG scores are highly correlated with average USMLE scores. Of the 3 USMLEs, Step 1 scores were most highly correlated with CREOGs. The inter-correlation for each of the USMLE exams individually and for CREOGs is shown. *** signifies p ≤ .001. (n = 75).

Figure 4.

CREOG regression analysis summary. Stepwise linear regressions were run to establish a relationship between CREOG performance and USMLEs. Step 1 was highly related with CREOGs and the loss of this metric impacts predicting applicant’s performance. Unshaded rows include Step 1, while shaded rows omit Step 1, and dark-shaded rows add additional metrics. R² show that it is possible to retain predictive ability after the loss of Step 1 if other metrics, such as MSPE wording or class rank are included.

Figure 5.

Regression analysis. Setting passing score on USMLE Step 1 to 209 is predictive of performance on CREOGs. Regression analysis shows that Step 1 was highly related with CREOGs. A Step 1 score >209 was predictive of average CREOG scores of 199. Since the CREOG exam is scored with a mean of 200 and a standard deviation of 20 for all test-takers, setting the passing score for Step 1 to 209 would reassure program directors that the candidate will do well on the CREOG exam (n = 75).