Instruments to evaluate non-technical skills during high fidelity simulation: A systematic review

Abstract

Introduction: High Fidelity Simulations (HFS) are increasingly used to develop Non-Technical Skills (NTS) in healthcare providers, medical and nursing students. Instruments to measure NTS are needed to evaluate the healthcare providers' (HCPs) performance during HFS. The aim of this systematic review is to describe the domains, items, characteristics and psychometric properties of instruments devised to evaluate the NTS of HCPs during HFS.

Methods: A systematic review of the literature was performed according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Studies were retrieved from PubMed, Cinahl, Web of Science, Cochrane Library, ProQuest and PubPsych. Studies evaluating the measurement properties of instruments used to assess NTS during HFS training were included. Pairs of independent reviewers determined the eligibility, extracted and evaluated the data. Risk of bias and appraisal of the methodological quality of the studies was assessed using the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist, and the quality of the evidence with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE).

Results: A total of 3,953 articles were screened. A total of 110 reports were assessed for eligibility and 26 studies were included. Studies were conducted in Europe/United Kingdom (n = 13; 50%), North America/Australia (n = 12; 46%) and Thailand (n = 1; 4%). The NTS instruments reported in this review included from 1 to 14 domains (median of 4, Q₁ = 3.75, Q₃ = 5) and from 3 to 63 items (median of 15, Q₁ = 10, Q₃ = 19.75). Out of 19 NTS assessment instruments for HFS, the Team Emergency Assessment Measure (TEAM) can be recommended for use to assess NTS. All the other instruments require further research to assess their quality in order to be recommended for use during HFS training. Eight NTS instruments had a positive overall rating of their content validity with at least a moderate quality of evidence.

Conclusion: Among a large variety of published instruments, TEAM can be recommended for use to assess NTS during HFS. Evidence is still limited on essential aspects of validity and reliability of all the other NTS instruments included in this review. Further research is warranted to establish their performance in order to be reliably used for HFS.

Keywords: assessment and evaluation; crew resource management; high fidelity simulation; human error; non-technical skills; psychometrics; reproducibility of results; teamwork.

FIGURE 1

PRISMA Flow diagram of article screening and selection process.

FIGURE 2

Quality assessment. The assessed measurement properties of 19 NTS instruments to which the plotted lines refer, are distributed around the radar. The red line in the first chart plots the proportion of studies with a very good or adequate methodological quality, over a total of 26 studies. The green line in the second chart represents the proportion of instruments with good measurement properties. The blue line in the third chart shows the proportion of instruments with high or moderate evidence quality. For example, content validity was of very good or adequate methodological quality in 27% of the included articles (3/11 studies), instruments with good measurement properties of content validity were 42% (8/19 instruments) and instruments with high or moderate evidence quality of content validity were 42% (8/19 instruments).