Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

Affiliations

¹ Y.S. Park is head, Department of Medical Education, and The Ilene B. Harris Endowed Professor, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0001-8583-4335.
² M.S. Ryan is associate dean for assessment, evaluation, research, and innovation, and professor of pediatrics, University of Virginia School of Medicine, Charlottesville, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289.
³ S.O. Hogan is director of outcomes research and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0009-0008-9006-1857.
⁴ K. Berg is associate dean of assessment, director, Rector Clinical Skills and Simulation Center, and professor of medicine, Sydney Kimmel Medical College, Philadelphia, Pennsylvania.
⁵ A. Eickmeyer is director of medical school education, University of Chicago Pritzker School of Medicine, Chicago, Illinois, and a PhD candidate, Maastricht University School of Health Professions Education, Maastricht, the Netherlands.
⁶ T.L. Fancher is associate dean for workforce innovation and education quality improvement and professor of medicine, University of California, Davis, School of Medicine, Sacramento, California.
⁷ J. Farnan is associate dean for undergraduate medical education and professor of medicine, University of Chicago Pritzker School of Medicine, Chicago, Illinois.
⁸ L. Lawson is senior associate dean of medical education and student affairs and professor of emergency medicine, Virginia Commonwealth University, Richmond, Virginia.
⁹ L. Turner is assistant dean for evaluation and assessment and assistant professor of medical education, The University of Cincinnati College of Medicine, Cincinnati, Ohio.
¹⁰ M. Westervelt is director of educational assessment, scholarship, improvement, and innovation, Office of Medical Education, University of California, Davis, School of Medicine, Sacramento, California.
¹¹ E. Holmboe is chief, research, milestones development and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois.
¹² S.A. Santen is senior associate dean, Virginia Commonwealth University, Richmond, Virginia, and professor, emergency medicine and medical education, University of Cincinnati, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-8327-8002.

PMID: 37983405
DOI: 10.1097/ACM.0000000000005366

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

Yoon Soo Park et al. Acad Med. 2023.

. 2023 Nov 1;98(11S):S123-S132.

doi: 10.1097/ACM.0000000000005366. Epub 2023 Aug 3.

Affiliations

¹ Y.S. Park is head, Department of Medical Education, and The Ilene B. Harris Endowed Professor, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0001-8583-4335.
² M.S. Ryan is associate dean for assessment, evaluation, research, and innovation, and professor of pediatrics, University of Virginia School of Medicine, Charlottesville, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289.
³ S.O. Hogan is director of outcomes research and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0009-0008-9006-1857.
⁴ K. Berg is associate dean of assessment, director, Rector Clinical Skills and Simulation Center, and professor of medicine, Sydney Kimmel Medical College, Philadelphia, Pennsylvania.
⁵ A. Eickmeyer is director of medical school education, University of Chicago Pritzker School of Medicine, Chicago, Illinois, and a PhD candidate, Maastricht University School of Health Professions Education, Maastricht, the Netherlands.
⁶ T.L. Fancher is associate dean for workforce innovation and education quality improvement and professor of medicine, University of California, Davis, School of Medicine, Sacramento, California.
⁷ J. Farnan is associate dean for undergraduate medical education and professor of medicine, University of Chicago Pritzker School of Medicine, Chicago, Illinois.
⁸ L. Lawson is senior associate dean of medical education and student affairs and professor of emergency medicine, Virginia Commonwealth University, Richmond, Virginia.
⁹ L. Turner is assistant dean for evaluation and assessment and assistant professor of medical education, The University of Cincinnati College of Medicine, Cincinnati, Ohio.
¹⁰ M. Westervelt is director of educational assessment, scholarship, improvement, and innovation, Office of Medical Education, University of California, Davis, School of Medicine, Sacramento, California.
¹¹ E. Holmboe is chief, research, milestones development and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois.
¹² S.A. Santen is senior associate dean, Virginia Commonwealth University, Richmond, Virginia, and professor, emergency medicine and medical education, University of Cincinnati, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-8327-8002.

PMID: 37983405
DOI: 10.1097/ACM.0000000000005366

Abstract

Purpose: The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM).

Method: Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings.

Results: During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS).

Conclusions: The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

PubMed Disclaimer

Cited by

Program Effects as a Source of Construct-Irrelevant Variance in ACGME Milestone Ratings.
Hamstra SJ, Yamazaki K. Hamstra SJ, et al. J Grad Med Educ. 2025 Aug;17(4):434-438. doi: 10.4300/JGME-D-24-00615.1. Epub 2025 Aug 15. J Grad Med Educ. 2025. PMID: 40832074 Free PMC article. No abstract available.
Program Directors' Assessments of US Medical Graduates' Transition to Residency.
Grbic D, Andriole DA, Roskovensky L, Speicher M, Horvath KA, Howley L. Grbic D, et al. JAMA Netw Open. 2025 Jan 2;8(1):e2454048. doi: 10.1001/jamanetworkopen.2024.54048. JAMA Netw Open. 2025. PMID: 39786772 Free PMC article.
Variability in Learner Performance Using the ACGME Harmonized Milestones During the First Year of Postgraduate Training.
Santen SA, Ryan MS, Fancher TL, Carcamo T, Hogan SO, Turner L, Cheung JJH, Berg K, Feldman M, Holmboe ES, Park YS; Medical School Outcomes Milestone Study Group. Santen SA, et al. Acad Med. 2025 Jul 1;100(7):852-859. doi: 10.1097/ACM.0000000000006053. Epub 2025 Apr 1. Acad Med. 2025. PMID: 40168552 Free PMC article.

References

1. Coalition for Physician Accountability. The Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC): Recommendations for comprehensive improvement of the UME-GME transition. https://physicianaccountability.org/wp-content/uploads/2021/08/UGRC-Coal.... Accessed December 7, 2022.
1. Germann CA, Strout TD, Park YS, Tekian A. Senior-year curriculum in U.S. medical schools: A scoping review. Teach Learn Med. 2020;32:34–44.
1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system: Rationale and benefits. N Engl J Med. 2020;366:1051–1056.
1. Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency. https://store.aamc.org/downloadable/download/sample/sample_id/63/%20. Accessed December 7, 2022.
1. Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency: Summary of the 10-school pilot (2014-2021). https://store.aamc.org/downloadable/download/sample/sample_id/554. Accessed December 7, 2022.

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- Ovid Technologies, Inc.
- Wolters Kluwer

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

Affiliations

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

Authors

Affiliations

Abstract

Similar articles

Cited by

References

MeSH terms

LinkOut - more resources

Full Text Sources