Development and Validity of a Simulation Program for Assessment of Clinical Teaching Skills

Abstract

Background: Teaching competence is expected of all intensivists, yet experts rarely supervise or assess trainees' teaching skills. Simulation offers an attractive solution. Objective: Develop and validate a simulation-based assessment of clinical teaching skills in pediatric critical care medicine (CCM). Methods: Participants were 128 pediatric CCM trainees, registered nurses, and respiratory therapists. Medical education experts used literature review and consensus to design three scenarios to assess teaching skills. Scenarios were piloted before use, and raters were trained. Teams completed one of three teaching scenarios, followed by a communication scenario. Raters were faculty members and trainees. Evidence for validity was collected and analyzed using Messick's unifying framework under the following domains: content, response processes, internal structure, relationship to other variables, and consequences of the assessment. Results: The scenarios and assessment tools were designed to capture the characteristics of a good teacher as described in the literature. Raters provided feedback that the tools were easy to use. Internal consistency of the scores measured by Cronbach's α was high. Rater agreement measured by interclass correlation was moderate for one of three scenarios. The relationship to other variables was investigated by correlating teaching scores with communication. Pearson's correlation was moderate for two of three scenarios. Consequences evidence was gathered using a retrospective self-assessed learning gain before versus after the training, which was significant for all scenarios. Conclusion: We developed a three-station simulation program for the assessment of teaching skills in pediatric CCM. The validity evidence collected is moderate, which indicates that it is effective for training and feedback on teaching skills.

Keywords: assessment; feedback; simulation; teaching; validity.

Figure 1.

The process of developing assessment of teaching skills through simulation and collecting validity evidence. PCCM = pulmonary critical care medicine.