Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: a multicenter study

Abstract

Background: The US Medical Licensing Examination (USMLE) and Orthopaedic In-Training Examination (OITE) are commonly used to select medical students or residents, respectively. Knowing how well these examinations predict performance on the American Board of Orthopaedic Surgery (ABOS) Part I certifying examination is important to provide evaluations for medical students and residents. Previous studies comparing the OITE scores with the ABOS Part 1 scores have been limited to one program.

Questions/purposes: Therefore, we compared the scores on the USMLE Step 1 and OITE examinations with those on the ABOS Part I certifying examination using data from four ACGME approved residency programs.

Methods: We reviewed 202 resident files from 1996 to 2008 from four programs in the same geographic region. Of those, 181 (90%) had complete records. De-identified data were used to compare USMLE Step 1 scores, OITE percentile rank scores, and ABOS Part I percentile rank scores. Pearson coefficients and receiver operator curves were calculated to assess the relationships between tests.

Results: We found a correlation of 0.53 between the USMLE Step 1 and ABOS Part I, and an average correlation of 0.50 for postgraduate years (PGY) 2 through 5 OITE scores and ABOS Part I. There was a stepwise increase in correlation from PGY 2 through PGY 5 between the OITE scores and ABOS scores. Those who averaged in the 27th percentile or lower on the OITE had a 57% chance of failing the ABOS Part I examination.

Conclusion: USMLE Step 1 scores correlated with ABOS Part I certifying examination scores, and we therefore believe it may be used as one factor in resident selection. Use of the OITE scores in guiding education and feedback appears to be justified.

Fig. 1

A ROC curve comparing the mean percentile rankings of the OITE examination scores for PGY 2 through PGY 5 with the ABOS Part I certifying examination scores is shown. If the mean OITE scores were in the 27th percentile or less, 57% failed the ABOS examination. If the scores were in the 37th percentile or less, the failure rate was 32%.

Fig. 2

A ROC curve comparing USMLE scores with ABOS Part I certifying examination scores is shown. For residents scoring 209 or less on the USMLE, 33% failed the ABOS examination.

Fig. 3

A ROC curve comparing the OITE percentile ranking of PGY 5 residents with the ABOS Part I certifying examination scores is shown. If the scores were at or below the 22nd percentile, 54% of the candidates failed the ABOS examination. If their scores were at or below the 40th percentile, 32% failed the examination.