A comparison of standardized and narrative letters of recommendation

Abstract

Objective: To compare the Council of Emergency Medicine Residency Directors' (CORD's) standardized letters of recommendation (SLORs) with traditional narrative letters of recommendation (NLORs) with regard to interrater reliability, consistency, and time of interpretation.

Methods: In part I of the study, four members of the residency selection committee each evaluated the same 20 SLORs and 20 NLORs from which all identifying characteristics had been deleted. Using Likert-type scales of the global assessment, each letter was assigned a numeric value from 1 to 7. The interrater reliability was calculated for both types of letters using the Kendall coefficient of concordance. Average time to interpretation of the letters was also determined. In part II, using the same numeric values as in part I, 207 single-author SLOR/NLOR pairs were evaluated to determine whether the global assessment of the SLOR was consistent with that of its partner NLOR. Interpretation of the NLOR was performed blinded to the SLOR. Statistical analysis was calculated using Spearman correlation coefficients.

Results: In part I of the study, the interrater reliability of the SLOR was 0.97, as compared with 0.78 for the NLOR. The average time to interpret the global assessment of the SLOR was 16 seconds, vs 90 seconds for the NLOR. In part II of the study, of the 207 SLOR/NLOR pairs, 112 (54%) were assigned the same numeric value, 80 (39%) differed by one, 13 (6%) differed by two, and two (1%) differed by three, for an overall correlation of 0.58.

Conclusions: Compared with NLORs, the CORD SLOR offers better interrater reliability with less interpretation time. Single-author SLOR/NLOR pairs submitted for a single applicant do not correlate well. Residency selection committees must decide whether the added work of interpreting NLORs is beneficial.