Education Research: unsatisfactory NEX rating correlations: searching for the reasons

Abstract

Objectives: To determine whether the previously demonstrated poor correlation between local faculty and external American Board of Psychiatry and Neurology (ABPN) examiners evaluating the Neurology Evaluation Exercise (NEX) is attributable to a difference between raters who know the residents and raters who do not, a difference between raters with ABPN experience and raters without it, or some other factor.

Methods: Deidentified NEX encounters were videotaped at 2 neurology residency programs. Each video was graded by 1 local faculty examiner, 1 external faculty examiner with ABPN experience, and 1 external faculty examiner without ABPN experience, using the ABPN-sanctioned form. Acceptable/unacceptable rates were compared using Cohen κ statistic.

Results: Fifty-eight videotaped NEX encounters involving 20 residents were evaluated by 12 local faculty examiners, 13 ABPN examiners, and 10 external non-ABPN examiners. The level of agreement between groups failed to meet our target κ of 0.7 (ABPN vs non-ABPN external examiners: κ = 0.47 [95% confidence interval 0.21-0.73]; local vs non-ABPN external examiners: κ = 0.37 [95% confidence interval 0.08-0.66]; local vs ABPN external examiners: κ = 0.40 [95% confidence interval 0.14-0.67]). Local, non-ABPN, and ABPN examiners assigned a failing grade to 13 (22%), 11 (19%), and 16 (28%) of the NEX encounters, respectively.

Conclusions: The disappointing correlation between local examiners, non-ABPN external examiners, and ABPN external examiners is not solely attributable to bias toward familiar residents. Inadequate training in NEX administration and scoring could be a factor. It is also possible that the NEX is not a valid tool. Further study is necessary.