Psychometric properties of an instrument designed to measure the educational quality of graduate training programs

Abstract

To assess the quality of residency education programs at an academic medical center for purposes of enhancing individual graduate medical education programs, we asked residents and fellows (N = 419) to evaluate their training programs using a Web-based questionnaire (response rate = 70%). Kruskal-Wallis tests, factor analysis, correlations, generalizability/decision studies, and mean plots were used to examine trainee responses and to assess the questionnaire's measurement properties. Exploratory factor analysis indicated that the instrument had a three-factor structure that correlated highly with overall program rating. Cronbach's alpha exceeded .80 for all factors, and decision studies revealed that 13 to 23 raters were needed to obtain G-coefficients greater than .70. Mean plots showed that the instrument could discriminate within and among training programs at the item level and the factor level, which should help target improvements across graduate training programs within large institutions.