Development of a Rigorously Designed Procedural Checklist for Assessment of Emergency Medicine Resident Performance of Temporary Transvenous Cardiac Pacing

Abstract

Objectives: Temporary transvenous cardiac pacing (TVP) is a potentially lifesaving intervention included in the list of essential core procedures for emergency medicine (EM) training; however, opportunities to perform TVP during residency cannot be guaranteed. EM graduates report feeling subjectively underprepared for this procedure, but objective performance data are lacking. Checklist-based simulated assessment is an increasing focus of competency-based medical education, particularly for invasive procedures like TVP. The objectives of this paper were as follows: first, to enlist a multidisciplinary team of experts to create an assessment tool for TVP using best practices in checklist development; second, to determine the reliability of checklist scoring; and third, to assess EM residents' baseline ability to perform TVP using a dedicated task trainer.

Methods: This study was conducted at a single 4-year EM residency. A panel of emergency physicians and cardiologists designed a TVP checklist using a modified Delphi approach. After consensus was achieved on a final checklist, EM residents were assessed using a dedicated TVP task trainer. Inter-rater reliability was determined using Cohen's kappa coefficient. Resident performance was determined by number of correctly performed checklist items.

Results: The expert panel achieved consensus on a 30-item checklist after three rounds of revisions. The Cohen's kappa coefficient for the overall checklist score was 0.87, with individual checklist items ranging from 0.63 to 1.00. In total, 58 residents were assessed with a mean score of 13.5 of 30 checklist items. Scores increased with each year of training.

Conclusions: This study details the rigorous development of a TVP checklist designed by a multidisciplinary team of experts. Checklist scores demonstrated strong inter-rater reliability. The overall poor performance of this cohort suggests the current approach to TVP training does not provide sufficient preparation for EM residents. Competency-based techniques, such as simulation-based mastery learning, should be explored.

Figure 1

Emergency medicine resident TVP checklist performance by PGY class. Box limits represent the 25th and 75th percentiles with the median checklist score represented by the bar. PGY = postgraduate year; TVP = transvenous cardiac pacing.