Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

Abstract

Importance: Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.

Objective: To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.

Design, setting, and participants: This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.

Interventions: Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.

Main outcomes and measures: To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.

Results: This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.

Conclusions and relevance: These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Figure 1.. Milestones Levels by Core Competency: Box Plots by Reporting Period for 3872 Residents in 514 Programs

Box plots use data for all subcompetencies within the Core Competency. The median is denoted by the line within the box; 25th percentile, bottom border of box; 75th percentile, top border of box; variability outside the IQR, whiskers; and outside values, dots. ICS indicates interpersonal and communication skills; MK, medical knowledge; PBLI, practice-based learning and improvement; PC, patient care; and SBP, system-based practice.

Figure 2.. Growth Trajectories for Practice-Based Learning and Improvement–3 (PBLI-3) and Systems-Based Practice–2 (SBP-2)

PBLI-3 is improves systems in which the physician provides care; SBP-2, emphasizes patient safety. Learning trajectories reflect uniquely distinct pathways of learners, as identified from the national data. Learners in group 3 did not achieve the level 4 graduation target indicating that they are ready for unsupervised practice.

Figure 3.. Growth Trajectories for Professionalism-2 (PROF-2) and Interpersonal and Communication Skills–4 (ICS-4)

PROF-2 is demonstrates professional conduct and accountability; ICS-4, utilizes technology to optimize communication. Learning trajectories reflect uniquely distinct pathways of learners, as identified from the national data. Learners in group 3 did not achieve the level 4 graduation target indicating ready for unsupervised practice.