Inaccuracy of the global assessment score in the emergency medicine standard letter of recommendation

Abstract

Objectives: The standard letter of recommendation (SLOR) is used by most emergency medicine (EM) faculty to submit evaluations for medical students applying for EM residency programs. In the global assessment score (GAS) section, there is a crucial summative question that asks letter writers to estimate the applicant's rank order list (ROL) position in their own program. The primary aim of the study was to determine if these estimated global assessment tiers agreed with the actual ROL, using the criteria recommended by the Council of Emergency Medicine Residency Directors (CORD).

Methods: Data from SLORs written by EM faculty from five California institutions were retrospectively collected from the 2008-2009 residency application year. Descriptive and comparative statistical analyses were performed using the documented GAS tiers and actual ROL positions.

Results: A total of 105 SLORs were reviewed from the five participating institutions. Three SLORs were excluded and 102 were analyzed. Only 27 (26%) SLORs documented a GAS tier that accurately predicted the applicant's actual ROL position. The GAS tier overestimated the applicant's position on the ROL in 67 (66%) SLORs, whereas it underestimated the position in eight (8%) SLORs. Accuracy was poor regardless of the number of letter writers on the SLOR (p = 0.890), the writer's administrative title (p = 0.326), whether the student was a home or visiting student (p = 0.801), or if the student had prior EM rotation experience (p = 0.793).

Conclusions: Standard letter of recommendation writers are inaccurate in estimating the ROL position of the applicant using the GAS tier criteria. The GAS tiers were accurate only 26% of the time. Because of the valuable role that the SLOR plays in determining an applicant's competitiveness in the National Resident Matching Program (NRMP) in EM, future discussion should focus on improving the consistency and accuracy of the GAS section. Furthermore, there needs to be a national dialogue to reassess the utility of the criterion-based GAS within the SLOR.