Is USMLE Step 1 score a valid predictor of success in surgical residency?

Abstract

Background: Many programs rely extensively on United States Medical Licensing Examination (USMLE) scores for interviews/selection of surgical residents. However, their predictive ability remains controversial. We examined the association between USMLE scores and success in surgical residency.

Methods: We compared USMLE scores for 123 general surgical residents who trained in the past 20 years and their performance evaluation. Scores were normalized to the mean for the testing year and expressed as a ratio (1 = mean). Performances were evaluated by (1) rotation evaluations; (2) "dropouts;" (3) overall American Board of Surgery pass rate; (4) first-time American Board of Surgery pass rate; and (5) a retrospective comprehensive faculty evaluation. For the latter, 16 surgeons (average faculty tenure 22 years) rated residents on a 1 to 4 score (1 = fair; 4 = excellent).

Results: Rotation evaluations by faculty and "drop out" rates were not associated with USMLE score differences (dropouts had average above the mean). One hundred percent of general surgery practitioners achieved board certification regardless of USMLE score but trainees with an average above the mean had a higher first-time pass rate (P = .04). Data from the comprehensive faculty evaluations were conflicting: there was a moderate degree of correlation between board scores and faculty evaluations (r = .287, P = .001). However, a score above the mean was associated with a faculty ranking of 3 to 4 in only 51.7% of trainees.

Conclusion: Higher USMLE scores were associated with higher faculty evaluations and first-time board pass rates. However, their positive predictive value was only 50% for higher faculty evaluations and a high overall board pass rate can be achieved regardless of USMLE scores. USMLE Step 1 score is a valid tool for selecting residents but caution might be indicated in using it as a single selection factor.

Keywords: Evaluation; Predictive value; Surgical training; USMLE.