An Automated System for Physician Trainee Procedure Logging via Electronic Health Records

Abstract

Importance: Procedural proficiency is a core competency for graduate medical education; however, procedural reporting often relies on manual workflows that are duplicative and generate data whose validity and accuracy are difficult to assess. Failure to accurately gather these data can impede learner progression, delay procedures, and negatively impact patient safety.

Objective: To examine accuracy and procedure logging completeness of a system that extracts procedural data from an electronic health record system and uploads these data securely to an application used by many residency programs for accreditation.

Design, setting, and participants: This quality improvement study of all emergency medicine resident physicians at University of California, San Diego Health was performed from May 23, 2023, to June 25, 2023.

Exposures: Automated system for procedure data extraction and upload to a residency management software application.

Main outcomes and measures: The number of procedures captured by the automated system when running silently compared with manually logged procedures in the same timeframe, as well as accuracy of the data upload.

Results: Forty-seven residents participated in the initial silent assessment of the extraction component of the system. During a 1-year period (May 23, 2022, to May 7, 2023), 4291 procedures were manually logged by residents, compared with 7617 procedures captured by the automated system during the same period, representing a 78% increase. During assessment of the upload component of the system (May 8, 2023, to June 25, 2023), a total of 1353 procedures and patient encounters were evaluated, with the system operating with a sensitivity of 97.4%, specificity of 100%, and overall accuracy of 99.5%.

Conclusions and relevance: In this quality improvement study of emergency medicine resident physicians, an automated system demonstrated that reliance on self-reported procedure logging resulted in significant procedural underreporting compared with the use of data obtained at the point of performance. Additionally, this system afforded a degree of reliability and validity heretofore absent from the usual after-the-fact procedure logging workflows while using a novel application programming interface-based approach. To our knowledge, this system constitutes the first generalizable implementation of an automated solution to a problem that has existed in graduate medical education for decades.

Figure 1.. Comparison of Automated and Manual Capture of Procedure Counts for Selected Procedures by Training Year

PGY indicates postgraduate year; POCUS, point-of-care ultrasonography.

Figure 2.. Procedure and Resuscitation Counts by Week, May 22, 2022, to May 7, 2023

For each modality, dotted lines represent the mean number of procedures logged per week, and the shaded area represents the 95% CIs. ^aFor the week of May 7, 2023, only 1 day of procedures were logged.