Validation of an Instrument for Real-Time Assessment of Neonatal Intubation Skills: A Randomized Controlled Simulation Study

Abstract

Objective: This study aimed to evaluate the construct validity and reliability of real-time assessment of a previously developed neonatal intubation scoring instrument (NISI).

Study design: We performed a randomized controlled simulation study at a simulation-based research and training facility. Twenty-four clinicians experienced in neonatal intubation ("experts") and 11 medical students ("novices") performed two identical elective intubations on a neonatal patient simulator. Subjects were randomly assigned to either the intervention group, receiving predefined feedback between the two intubations, or the control group, receiving no feedback. Using the previously developed NISI, all intubations were assessed, both in real time and remotely on video. Construct validity was evaluated by (1) comparing the intubation performances, expressed as percentage scores, with and without feedback, and (2) correlating the intubation performances with the subjects' level of experience. The intrarater reliability, expressed as intraclass correlation coefficient (ICC), of real-time assessment compared with video-based assessment was determined.

Results: The intervention group contained 18 subjects, the control group 17. Background characteristics and baseline intubation scores were comparable in both groups. The median (IQR) change in percentage scores between the first and second intubation was significantly different between the intervention and control group (11.6% [4.7-22.8%] vs. 1.4% [0.0-5.7%], respectively; p = 0.013). The 95% CI for this 10.2% difference was 2.2 to 21.4%. The subjects' experience level correlated significantly with their percentage scores (Spearman's R = 0.70; p <0.01). ICC's were 0.95 (95% CI: 0.89-0.97) and 0.94 (95% CI: 0.89-0.97) for the first and second intubation, respectively.

Conclusion: Our NISI has construct validity and is reliable for real-time assessment.

Key points: · Our neonatal intubation scoring instrument has construct validity.. · Our instrument can be reliably employed to assess neonatal intubation skills directly in real time.. · It is suitable for formative assessment, i.e., providing direct feedback during procedural training..