Measuring learning gain during a one-day introductory bronchoscopy course

Abstract

Background: Rigorous assessment of medical knowledge and technical skill inspires learning, reinforces confidence, and reassures the public. Identifying curricular effectiveness using objective measures of learning is therefore crucial for competency-oriented program development in a learner-centric educational environment. The aim of this study was to determine whether various measures of learning, including class-average normalized gain, can be used to assess the effectiveness of a one-day introductory bronchoscopy course curriculum.

Methods: We conducted a quasi-experimental one-group pre-test/post-test study at the University of California, Irvine. The group comprised 24 first-year pulmonary and critical care trainees from eight training institutions in southern California. Class-average normalized gain, single-student normalized gain, absolute gain, and relative gain were used as objective measures of cognitive knowledge and bronchoscopy technical skill learning. A class-average normalized gain of 30% was used to determine curricular effectiveness. Perceived educational value using Likert-scale surveys and post-course questionnaires was determined during and 3 months after course participation.

Results: Mean test scores of cognitive knowledge improved significantly from 48 to 66% (p = 0.043). Absolute gain for the class was 18%, relative gain was 37%, class average normalized gain <g> was 34%, and the average of the single-student normalized gains g(ave) was 29%. Mean test scores of technical skill improved significantly from 43 to 77% (p = 0.017). Absolute gain was 34%, relative gain was 78%, class average normalized gain <g> was 60%, and the average of the single-student normalized gains g(ave) was 59%. Statistically significant improvements in absolute gain were noted in all five elements of technical skill (p < 0.05). Likert-scale surveys, questionnaires, and surveys demonstrated strong perceived educational value.

Conclusion: The effectiveness of a one-day introductory bronchoscopy curriculum was demonstrated using a pre-test/post-test model with calculation of normalized gain and related metrics.

Fig. 1

Bronchoscopy technical skills pre-test and post-test score sheet

Fig. 2

A Cognitive learning pre- and post-test score plots showing improvement (positive slopes), no change (horizontal lines), or deterioration (negative slopes); 1 = pre-test, 2 = post-test. Maximum score was 20 for cognitive knowledge assessments. B Technical skill learning pre- and post-test score plots showing improvement (positive slopes), no change (horizontal lines) or deterioration (negative slopes); 1 = pre-test, 2 = post-test. Maximum score was 16 for technical skill assessments

Fig. 3

A Cognitive and technical skill average learning gains with pre- and post-test scores (%) during one-day introductory bronchoscopy course (N = 24). B Average learning gains of five components of bronchoscopy technical skill with pre- and post-test skill scores (%). All changes statistically significant (p < 0.05)