Development of a Standardized Assessment of Simulation-based Extracorporeal Membrane Oxygenation Educational Courses

Abstract

Background: In 2020, the Extracorporeal Life Support Organization education task force identified seven extracorporeal membrane oxygenation (ECMO) educational domains that would benefit from international collaborative efforts. These included research efforts to delineate the impact and outcomes of ECMO courses.

Objective: Development of a standardized online assessment tool to evaluate the effectiveness of didactic and simulation-based ECMO courses on participants' confidence, knowledge, and simulation-based skills; participant satisfaction; and course educational benefits.

Methods: We performed a prospective multicenter observational study of five different U.S. academic institution-based adult ECMO courses that met Extracorporeal Life Support Organization endorsement requirements for course structure, educational content, and objectives. Standardized online forms were developed and administered before and after courses, assessing demographics, self-assessment regarding ECMO management, and knowledge examination (15 simple-recall multiple-choice questions). Psychomotor skill assessment was performed during the course (time to complete prespecified critical actions during simulation scenarios). Self-assessment evaluated cognitive, behavioral, and technical aspects of ECMO; course satisfaction; and educational benefits.

Results: Out of 211 participants, 107 completed both pre- and postcourse self-assessment forms (97 completed both pre- and postcourse knowledge forms). Fifty-three percent of respondents were physician intensivists, with most (51%) practicing at academic hospitals and with less than 1 year of ECMO experience (50%). After the course, participants reported significant increases in confidence across all domains (cognitive, technical, and behavioral, P < 0.0001, 95% confidence interval [CI], 1.2-1.5; P < 0.0001, 95% CI, 2.2-2.6; and P = 0.002, 95% CI, 1.7-2.1, respectively) with an increase in knowledge scores (P < 0.001; 95% CI, 1.4-2.5). These findings were most significant in participants with less ECMO experience. There were also significant reductions in times to critical actions in three of the four scored simulation scenarios. The results demonstrated participants' satisfaction with most course aspects, with more than 95% expressing that courses met their educational goals.

Conclusion: We developed and tested a structured ECMO course assessment tool, demonstrating participants' self-reported benefit as well as improvement in psychomotor skill acquisition, course satisfaction, and educational benefits. Course evaluation is feasible and potentially provides important information to improve ECMO courses. Future steps could include national implementation, addition of questions targeting clinical decision making to further assess knowledge gain, and multilanguage translation for implementation in international courses.

Keywords: Extracorporeal Life Support Organization education task force; course assessment; education; extracorporeal membrane oxygenation; simulation.

Figure 1.

Participant enrollment diagram. CONSORT (Consolidated Standards of Reporting Trials) diagram of the participant enrollment process from five ELSO-endorsed courses from March 2019 through February 2020 (four general ECMO and one VV course) with 211 participants, 156 completed precourse demographics, 100 completed precourse knowledge tests, and 138 precourse self-assessments. For the postcourse assessments, 102 completed the knowledge tests, and 107 completed the self-assessments. In total, 107 participants completed both pre- and postcourse self-assessments, and 97 completed pre- and postcourse knowledge tests. ECMO = extracorporeal membrane oxygenation; ELSO = extracorporeal life support organization; VV = venovenous.

Figure 2.

Pre- and postcourse self-assessment results. Box-and-whisker plots of the pre- and postcourse self-assessment results. There was a significant increase in the reported postcourse self-assessments across all three tested categories: cognitive, technical, and behavioral aspects of ECMO management (Wilcoxon signed rank sum test). CI = 95% confidence interval; Post = postcourse; Pre = precourse.

Figure 3.

Pre- and postcourse knowledge assessment tests. (A) Box-and-whisker plots of the pre- and postcourse knowledge assessment tests. There was no significant difference in either pre- or postcourse test results between the participant subgroups (Kruskal-Wallis one-way analysis of variance). There was a significant increase in the postcourse scores for all participants, in addition to the subgroups of intensivists, nonintensivists, physicians, and nonphysicians (Wilcoxon signed rank sum test), with a moderate to large effect size by Cohen’s d calculation for all groups. (B) Box-and-whisker plots of the pre- and postcourse knowledge assessment tests, by years of clinical experience. There was no significant difference in either pre- or postcourse test results in subgroups by clinical experience. There was a significant increase in the postcourse test results for both groups of participants with clinical experience less or more than 5 years (Wilcoxon signed rank sum test). There was a higher effect size in participants with less than 5 years of clinical experience than in those with more than 5 years of clinical experience (Cohen’s d test). (C) Box-and-whisker plot of the pre- and post-knowledge assessment test scores by groups according to ECMO experience. There was no significant difference in the pre- or postcourse test results, but there was a significant increase in the postcourse results for both groups, regardless of previous ECMO experience (Wilcoxon signed rank sum test). We found a larger effect size in the participants with less than 1 year of ECMO experience versus those with more than 1 year of ECMO experience (Cohen’s d). CI = 95% confidence interval; Post = postcourse; Pre = precourse.