Evaluation of the simulation based training quality assurance tool (SBT-QA10) as a measure of learners' perceptions during the action phase of simulation

Abstract

Background: In an earlier interview-based study the authors identified that learners experience one or more of eight explicit perceptual responses during the active phase of simulation-based training (SBT) comprising a sense: of belonging to instructor and group, of being under surveillance, of having autonomy and responsibility for patient management, of realism, of an understanding of the scenario in context, of conscious mental effort, of control of attention, and of engagement with task. These were adapted into a ten-item questionnaire: the Simulation Based Training Quality Assurance Tool (SBT-QA10) to allow monitoring of modifiable factors that may impact upon learners' experiences. This study assessed the construct validity evidence of the interpretation of the results when using SBT-QAT10.

Materials and methods: Recently graduated doctors and nurses participating in a SBT course on the topic of the deteriorating patient completed the SBT-QAT10 immediately following their participation in the scenarios. The primary outcome measure was internal consistency of the questionnaire items and their correlation to learners' satisfaction scores. A secondary outcome measure compared the impact of allocation to active versus observer role.

Results: A total of 349 questionnaires were returned by 96 course learners. The median of the total score for the ten perception items (TPS) was 39 (out of 50), with no significant difference between the scenarios. We identified fair and positive correlations between nine of the 10 items and the SBT-QA10-TPS, the exception being "mental effort". Compared to observers, active learners reported significantly more positive perceptions related to belonging to the team and interaction with the instructor, their sense of acting independently, and being focused. The questionnaire items were poorly correlated with the two measures of global satisfaction.

Conclusion: Except for the item for mental effort, the QA10-TPS measures learners' experiences during the active phase of simulation scenarios that are associated with a positive learning experience. The tool may have utility to learners, instructors, and course providers by informing subsequent debriefing and reflection upon practice for learners and faculty. The relationship between these perceptions and commonly used measures of satisfaction remains poorly understood raising questions about the value of the latter.

Keywords: Construct validity; Learners’ perception; Observer role; Quality Assurance Tool; Questionnaire development; Simulation-based learning; Simulation-based training.